We describe the method used to create a dataset of open-source safety-critical software, such as that used for autonomous cars, healthcare, and autonomous aviation, through a systematic and rigorous selection process. The dataset can be used for empirical studies regarding the quality assessment of safety-critical software, its dependencies, and its development process, as well as comparative studies considering software from other domains.
Safety-critical systems (SCS) are those whose failure could result in loss of life, significant property damage, or damage to the environment [1]. Over the past decades ever more software is developed and released as open-source software (OSS) -with licenses that allow its free use, study, change, and distribution [2]. The increasing adoption of open-source software in safety-critical systems [3], such as those used in the medical, aerospace, and automotive industries, poses an interesting challenge. On the one hand, it shortens time to delivery and lowers development costs [4]. On the other hand, it introduces questions regarding the system's quality. For a piece of software to be part of a safety-critical application it requires quality assurance, because quality is a crucial factor of an SCS's software [5]. This assurance demands that evidence of OSS quality is supplied, and an analysis is needed to assess if the certification cost is worthwhile.
Despite the increased interest by both industry and the research community for the adoption of OSS in SCS, there seems to be little evidence regarding the quality of such systems. An earlier study [3] explored the field through a systematic literature survey. It identified 22 relevant studies and isolated 31 software systems. Given that many OSS projects lack academic publications associated with them, we approach the question of finding OSS associated with SCS by looking directly for relevant software systems.
Envelope t8160018@aueb.gr (R. Galanopoulou); dds@aueb.gr (D. Spinellis) GLOBE https://www.dmst.aueb.gr/dds/ (D. Spinellis) Orcid 0000-0002-5318-9017 (R. Galanopoulou); 0000-0003-4231-1897 (D. Spinellis)
Our search is based on the methodology guidelines proposed by Kitchenham and Charters [6]. The software selection method consists of three steps:
• candidate system identification based on Google queries, GitHub tags queries, and results of other studies, • project repository identification and exclusion based on selection criteria, and • filtering based on software purpose and characteristics.
To retrieve relevant SCS open-source projects, we run by hand the following search queries. The provided search queries are notional; the disjunctions were performed by combining by hand the results of individual queries.
OR "open source hardware") We used this query in combination with the ones detailed in the following paragraphs, which target specific SCS domains, applications, and standards. By using "github" and "git repository" as search keywords, the OSS query finds projects that are likely to have a Git repository, which can then be queried through an API to further narrow down the selected projects. We added the "open source hardware" term based on the assumption that an open-source hardware system might use OSS as well.
SCS Domains: OSS AND ("Infrastructure" OR "Medicine" OR "Nuclear engineering" OR "Recreation" OR "Transport" OR "Railway" OR "Automotive" OR "Aviation" OR "Space flight") This query associated with broad SCS domains identified 15 projects in total (through a Google search and a GitHub tag search). We complemented these results for the domain of health applications by adding five projects derived from the data associated with a recent related study [7].
OSS AND ("Airbag systems" OR "Braking systems" OR "Seat belts" OR "Power Steering systems" OR "Advanced driver-assistance systems" OR "Electronic throttle control" OR "Battery management system for hybrids and electric vehicles" OR "Electric park brake" OR "Shift by wire systems" OR "Drive by wire systems" OR "Park by wire") We derived the terms of this query from Pimentel's book chapter [8].
The terms of this query were derived from Hamilton's [9] and Alemzadeh et al. [10] studies.
The query keywords were derived from studies conducted for civil infrastructure and systems used for emergency in buildings (e.g. fire) [11,12].
The three domain-specific queries identified 96 projects in total.
SCS Standards: OSS AND ("DO-178C" OR "MISRA" OR "MISRA-C" OR "IEC 61580" OR "IEC 880" OR "ISO 9000") The standard names were derived from conference presentations associated with the examined SCS domains derived from the Conference on Digital Avionics Systems [13]. This query identified 4 projects in total.
For practical reasons in each step we excluded projects lacking a description or having a non-English description (e.g. Japanese, Portuguese).
The second step of our selection method was based on a project's repository characteristics. Here we excluded projects without a Git repository (Github or GitLab), inactive projects (lacking a commit in the last eight months) and unpopular projects (having fewer than 70 GitHub stars). Through these criteria we rejected 57 projects leaving us with 63.
In our last step we applied an exclusion criterion based on whether the project served a safety-critical role. For example, in the medical category, we excluded projects associated with a hospital's enterprise resource planning (ERP) services or the keeping of patient records. We did this by running the following Google search query for each project and studying the results to identify the project's purpose. (project-name AND ("applications" OR "in open source hardware" OR "safety critical systems"). Through this criterion we rejected 21 projects leaving us with 42.
The selected projects are listed in Table 1. For each project we list its name, application field, popularity (in awarded stars), size (in thousands of lines of code), implementation language(s), and repository location. The whole dataset and replication package are available online. 1Most projects are in the automotive (AM) sector (12 out of 42), followed by ten projects in avionics (AV), six in medicine (MED), three in spaceflight (SP), two in nuclear engineering (NE), and one in recreation (RE).
Additionally, we found that 34 unique programming languages are used by the selected 42 projects. The most popular are C++ (used in 29 projects), C (25), Python (24), and the Unix shell (15).
The presented dataset suffers from some limitations, which could be lifted in the future. First, the threshold and inclusion criterion of 70 stars we used is arbitrary. Ideally it should be replaced by objective criteria, based e.g. on attributes that characterize engineered projects [14]. Second, the study has excluded projects hosted on platforms other than Github and Gitlab. This was done, because we are aiming to evaluate the dataset against community metrics that can be gathered from these platforms, such as the number of forked repositories, open and closed issues, and contributed pull requests. Nevertheless, if one does not care about these metrics, then more repository hosting platforms should be considered. Third, the dataset does not incorporate SCS testing software and components, which can be vital for satisfying safety requirements [15].
The development of SCS as OSS did not prove to be a widespread industrial practice, especially for avionics. We noticed that some of the field's leaders (BAE System Avionics, Furuno, Airbus Helicopters, GE Aviation, Telefunken) maintain GitHub organizations, but evidently prefer to keep their repositories private.
The low number of SCS OSS projects we found may be associated with onerous requirements and diminished incentives. Many SCS application domains, in common with accessibility OSS [16], may require either specialized peripherals (such as LIDAR or ECG sensors) or powerful hardware. In addition, SCS software may be based on specialized, inflexible, and non-free development tools [17]. These requirements, can hinder OSS developer participation. Furthermore, due to its specialized nature and regulatory requirements, SCS deployment may require strong organizational backing, thus further limiting and discouraging OSS volunteer participation.
Notably, none of the systems identified in the earlier study [3] passed our inclusion criteria. Most were excluded due to lack of activity or popularity in terms of GitHub stars. This is a concern regarding the long-term viability and maintenance of OSS SCSs.
The presented dataset can be used to evaluate the quality of OSS SCSs. This can be done e.g. based on the defect prediction model proposed by Foyzur and Premkumar [18], which outlines relevant process and code metrics. It is important to study both facets, because the most successful OSS projects are those featuring not only a high-quality code base, but also a thriving user community [4]. Results can then be compared with other OSS endeavors that have similar engineered project characteristics [14]. Furthermore, results can be qualitatively evaluated based on practices advocated by Wilson et al. [19,20].
This work has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825328.