GitHub Actions was introduced in 2019 as a software development workflow automation tool, allowing to automate a wide range of social and technical activities in GitHub repositories. The reusable components known as Actions are developed within GitHub repositories and can be distributed through the GitHub Marketplace. GitHub Actions forms an ecosystem because workflows can rely on reusable Actions, that themselves may depend on other components such as NodeJS packages, Docker images, or other Actions. Just as packages in software library ecosystems have been shown to sufer from a multitude of maintainability issues due to their complex dependency networks, we posit that the same is true for Actions. Therefore, this paper presents preliminary insights in the dependencies of Actions. Based on a dataset of 2,817 Actions, we report on the characteristics of these Actions and we explore to which extent they are developed using JavaScript, Docker or as composite Actions, and to which extent they depend on other components. We show that most Actions are developed using JavaScript, and that composite Actions are gradually replacing Docker Actions. We also show that Actions have many dependencies, especially towards JavaScript packages, resulting in a large number of deeply nested transitive dependencies. This justifies the need for further maintainability studies of the GitHub Actions ecosystem.
services, including automated building, testing, quality analysis, code review, communication, licence verOpen-source software (OSS) has gained significant popu- ification, and monitoring dependencies and security vullarity and typically forms a substantial portion of a mod- nerabilities. Only 18 months after its public release, ern software application stack, ranging from 70% to 90% GitHub Actions has become the dominant CI/CD serof its code base [1]. Software development has evolved vice on GitHub [6]. into a persistent, extensively distributed and collabora- GitHub Actions facilitates the creation of workflows by tive endeavour [2]. During collaborative development, a providing reusable Actions, that are distributed through multitude of tasks must be executed, including coding, de- the GitHub Marketplace.1 In January 2023, already over bugging, testing, quality and security analysis, packaging 17K Actions were available for reuse. There is a signifand releasing software distributions, and so forth. These icant amount of community involvement in the develtasks require the use of version control systems, soft- opment of these Actions, which entails a considerable ware distribution managers, bug and issue trackers, and risk of malicious actors that intentionally develop new quality, vulnerability and dependency analysers. Many or modify existing Actions to compromise software deof these tools are integrated into, or accessible through, velopment repositories. social coding platforms [3]. The largest such platform to Actions can be developed in three diferent ways: usdate is GitHub, hosting millions of software repositories ing JavaScript code, using Docker containers, or through and having served over 94 million users in 2022 [4]. the mechanism of composite Actions. Additionally, Ac
To speed up the pace of development while maintain- tions may depend on other Actions or on third-party ing high-quality software releases, continuous integra- software components such as npm packages or Docker tion, deployment, and delivery (CI/CD) was introduced images. Relying on such third-party components may to automate a plethora of repetitive development-related lead to maintainability issues such as incompatibilities, tasks [5]. With the introduction of GitHub Actions in security vulnerabilities, and the reliance on outdated or 2019, GitHub has integrated CI/CD support within GitHub abandoned components [7]. For instance, a study focusrepositories, providing direct access to a wide range of ing on the security vulnerabilities of composite Actions and JavaScript Actions revealed that around 30% of these Actions had at least one high or critical security alert in their dependencies [8]. This makes it crucial to analyse the dependencies of Actions. This justifies the current article, which provides a preliminary study of the dependency network of Actions that are developed through 1https://github.com/marketplace?type=actions GitHub repositories and published on the GitHub Mar- et al. [11] investigated the efects of GitHub Actions on ketplace. Such an investigation serves as an initial and GitHub projects. Valenzuela-Toledo and Bergel [12] confundamental stepping stone for researchers seeking to ducted a study on the usage and maintenance practices comprehend the potential impact of maintainability is- of GitHub Actions workflows in popular GitHub reposisues in the GitHub Actions ecosystem. tories and identified various types of workflow modifica
We carry out a quantitative empirical analysis with tions. Benedetti et al. [13] proposed a security assessment two main research goals. Goal G1 aims to characterise Ac- methodology to investigate the impact of security issues tions distributed through the GitHub Marketplace, quan- on GitHub Actions workflows and the software supply tifying the diferent approaches for developing Actions chain. Decan et al. [14] studied the usage of GitHub and their evolution over time. This goal is subdivided into Actions in GitHub repositories. They characterised the two research questions that are centered around a quan- repositories and their workflows by analysing job types, titative analysis of the characteristics of 2,817 Actions steps and used Actions. They observed that almost all on the GitHub Marketplace, as well as their 25,975 corre- workflows use Actions, which could potentially pose a sponding releases on the GitHub repositories in which problem as any issues with these Actions, including bugs, they are developed. security vulnerabilities, or outdated components, could 1.1 What are the characteristics of Actions distributed negatively impact the workflows that incorporated them. through the GitHub Marketplace? Saroar and Nayebi [15] investigated the motivations, 1.2 How do diferent types of Actions evolve? decision criteria, and challenges associated with creating,
Goal G2 focuses on the dependency characteristics publishing, and using GitHub Actions. They discovered for the three types of Actions, through the following that, when presented with comparable options, developresearch questions: ers tend to favour Actions created by verified individuals 2.1 What are the dependency characteristics of com- and having a higher number of stars. Furthermore, they posite Actions? noticed that users frequently switch to alternative Ac2.2 What are the dependency characteristics of Java- tions in response to issues such as bugs and insuficient Script Actions? documentation. Additionally, they found that config2.3 What are the dependency characteristics of Docker uring and debugging workflow files is one of the most Actions? common dificulties encountered by users of GitHub Actions. A recent study on security checks for Actions in the Marketplace also discovered that when Actions are 2. Related work forked and Dependabot 2 is enabled on the forked repositories, particularly for composite and JavaScript Actions, approximately 30% of these Actions contain at least one high or critical security alert in their dependencies [8].
To the best of our knowledge, our current quantitative study is the first to focus on how Actions are developed, diferentiating between diferent types of Actions and examining the characteristics, evolution and dependencies in the GitHub Actions ecosystem.
On the usage and adoption of GitHub Actions. Previous research on GitHub Actions has primarily focused on the usage and adoption of GitHub Actions, with limited investigation into their development. Several empirical studies have evaluated the adoption and usage details of Actions, typically by analysing data from GitHub repositories that use and adopt them for specific automation tasks. For example, Golzadeh et al. [6] analysed 91,810 GitHub repositories to study how the CI/CD landscape has changed since the introduction of GitHub Actions. They found that the adoption of Actions is associated with a decline of other CI/CD tools such as Travis, CircleCI, and Azure. Rostami et al. [9] conducted a qualitative study aimed at comprehending the underlying factors driving the transformations in the CI/CD landscape. In-depth interviews with 22 experienced software practitioners provided insights into their usage, co-usage, and migration patterns of 31 distinct CI/CD tools. Based on this qualitative analysis, they identified a discernible trend of migrating towards GitHub Actions, and they also pinpointed the primary drivers behind this trend.
Kinsman et al. [10] examined 3,190 repositories to investigate changes in various development activity indi- 2https://docs.github.com/en/code-security/ cators following the adoption of Github Actions. Chen dependabot/dependabot-security-updates/ about-dependabot-security-updates On the dependency management of software library ecosystems. Since this paper aims to investigate dependencies in the GitHub Actions ecosystem, it is worthwhile to explore prior studies that have explored dependencies in other software ecosystems.
It is a widely adopted practice among software developers to depend on reusable software components to benefit from pre-existing code, rather than creating everything from scratch [16]. To enhance this practice of reuse, package managers and registries of reusable libraries have been introduced for the predominant programming languages, such as npm for JavaScript, PyPI for Python, and Maven for Java. While software compo- oped in three diferent ways (i.e., it can be one of three nent reuse provides several benefits [ 17], it also comes types): with maintainability issues such as dependency manage- JavaScript Actions enable the execution of JavaScript ment [18, 19, 20], security vulnerabilities [21, 22, 23, 24, code within a Node.js runtime environment. They are 25, 26], compatibility issues [27, 28, 29], outdated compo- used for tasks requiring complex logic or interactions nents [30, 31], deprecated and obsolete components [32]. with the GitHub API or other external services. For exMirhosseini and Parnin [33] studied the incentives of ample, a JavaScript Action could be created to automate software developers to update their project dependen- the process of creating a new issue in a project managecies. The practice of maintaining outdated dependencies ment tool when a new pull request is opened. The use increases the risk of encountering bugs and security vul- of JavaScript also unlocks the potential to rely on a huge nerabilities. Cox et al. [34] analysed Java projects and number of JavaScript libraries and packages distributed found that projects relying on outdated dependencies through package managers such as npm. were four times more likely to experience security issues Docker Actions define tasks that are executed in a and compatibility problems. Docker container. This allows for greater flexibility and portability in workflow execution, as the environment can be customised to suit the needs of the workflow. A 3. About GitHub Actions Docker Action can be developed using a Dockerfile , which defines the base image of the container, and the various GitHub Actions is a CI/CD tool integrated into GitHub to commands that will be executed on top of this image. allow maintainers of GitHub repositories to automate a These base images can be found in container registries, wide range of tasks. Following the “configuration as code" such as Docker Hub, GitHub Container Registry, Google paradigm, workflows are specified as YAML files (stored Container Registry, Microsoft Container Registry, Red in the .github/workflows folder of the repository). A Hat Quay and Harbor. workflow reacts to one or more events (e.g., a pull request Composite Actions allow to combine multiple workis submitted, commits are pushed, or an issue is opened) lfow steps within one Action, similar to how jobs are and executes one or more jobs. Jobs are defined in terms defined in workflow files. They allow developers to simof the steps that will be executed when the workflow is plify complex workflows and reduce code duplication by triggered. A step can either define the shell commands defining such behaviour directly in a YAML file. that need to be executed (using the run: syntax) or can refer (using the uses: syntax) to a reusable component, called an Action, to carry out its task. 4. Data extraction process
The GitHub Actions Marketplace is a centralised platform that enables users to browse, discover, and share To study the three types of Actions and their dependenreusable Actions within the GitHub ecosystem. It serves cies, it is required to obtain a large dataset of Actions as a valuable resource for GitHub community to en- and their respective releases from GitHub repositories hance their productivity and streamline their workflows where the Actions were developed. In order to acquire through the use of reusable Action. It includes features such a dataset, we retrieved Actions that are distributed for filtering, sorting, and searching Actions based on var- through the GitHub Marketplace, which is the central ious criteria, such as category and popularity based on location for discovering reusable Actions. At the time of stars. In the GitHub Actions Marketplace, Actions can writing, there was no API to obtain all Actions distributed be assigned a primary and a secondary category. The on the Marketplace. Therefore, we gathered the Actions Actions that are published in the GitHub Marketplace contained in each category listed on the Marketplace. In typically include information such as the name of the Ac- order to exclude Actions that might have been created tion, its contributors, its primary and secondary category, only for personal or experimental purposes [35] we only a brief description of its functionality, information about considered Actions with at least 10 stars. the developer or organization that created the Action, the Through this process, we extracted 2,817 distinct Accorresponding GitHub repository, the number of stars, tions and their associated metadata including their prithe number of open issues, the number of pull requests, mary and secondary categories, the name of the reposiand the latest 10 releases,. tory in which the Action is developed, its number of stars,
Actions can be developed in any public GitHub reposi- issues, pull requests, and so on. Then, for each Action, we tory and shared on the GitHub Marketplace. To enable extracted its complete list of releases. To do so, we relied an Action to be reused, one has to create a YAML file on the GitHub API for Releases, and obtained the 25,975 named action.y(a)ml at the root of the repository. This releases that were created between November 2019 (the ifle details the metadata of the Action, such as its name, oficial release date of GitHub Actions) and January 2023 its set of parameters and its type. An Action can be devel- (the data extraction date).
Actions x10 releases Jan 2020
Jul
Jan 2021
Jul
Jul Jan 2022
Jan 2023
through the GitHub Marketplace and quantify the three types of Actions and how they have evolved over time. Through a quantitative empirical analysis of the 2,817 considered Actions and their 25,975 releases extracted from the corresponding GitHub repositories, we answer the two following research questions. 1.1: What are the characteristics of
defining their dependencies, this research question provides a comprehensive overview of the Actions in our dataset. Investigating the characteristics of Actions distributed through the GitHub Marketplace can provide valuable insights into the current landscape of Actions.
Table 1 reports on the median and mean values for several characteristics such as the number of stars, the number of releases, the age (in months) and the type of Actions. The first line (“all categories”) includes all Actions in our dataset. The next five lines list these characteristics for the top five categories comprising at least 5% of the Actions as either primary or secondary category. The last line corresponds to the Actions belonging to the remaining 18 categories. Notice that since an Action can belong to more than one category, the total number of Actions exceeds 2,817.
The categories of continuous integration, utilities, deployment, publishing and code quality are the most commonly used for publishing Actions on the GitHub Marketplace. In terms of popularity, the Actions in the utilities category had the highest average (157.5) and median (30) number of stars. We also found that the median and mean number of releases for Actions were 5 and 9.2, respectively. This indicates that Actions are still maintained and updated after their initial release. Regarding the type of Actions, JavaScript Actions and composite Actions are the most and least prevalent types of Actions, respectively, across the majority of categories. In the publishing and code quality categories, Docker Actions have a higher proportion than JavaScript Actions.
Overall, our observations did not reveal any significant diferences between the various categories of Actions. Approximately half of all analysed Actions had at least 27 stars, 5 releases and were developed for a duration of 3 years. A majority of the Actions were developed as JavaScript Actions, accounting for 52.7% of the total, followed by Docker Actions, which constituted 36.2% of the Actions and composite Actions, which made up the remaining 11.1%. However, these proportions difered across the distinct categories of Actions. 1.2: How do diferent types of Actions evolve? By analysing the YAML files of the latest releases of Actions in our dataset, we observed that 11.1%, 36.2%, and 52.7% of the considered Actions were released as composite, Docker, and JavaScript Actions, respectively. We also quantified the median and mean number of releases for the three types of Actions and observed that, although composite Actions is a new type of Action introduced in August 20203, they had more releases with a median of 5 and a mean of 9.4, as compared to Docker Actions with a median of 4 and a mean of 7.9.
Action developers may decide to change the type of Action over its lifetime. Therefore, investigating the migrations among the three diferent types of Actions and their evolution over the course of the observation period supplements the investigation of characteristics of Actions. Action developers may change the type of Action they are developing for a variety of reasons. The choice of Action type may depend on factors such as the required functionality, performance considerations, maintenance requirements and integration with other tools or services. Diferent types of Actions may be bet
composite-run-steps/ ter suited for diferent use cases. For example, a Docker reusability, customisability and extensibility. They enAction may be useful for running a containerised applica- able the definition of a set of steps that can be reused tion, while a JavaScript Action might be more appropriate across multiple workflows and repositories, facilitating for customising the behavior of a workflow. Diferent code reuse and reducing duplication. Moreover, regardtypes of Actions may also have varying performance ing the ability to combine shell commands and using characteristics. For example, a JavaScript Action may be Actions, they can be customised to specific workflows or faster than a Docker Action in some scenarios. Mainte- repositories and extended with additional functionality nance may also be a factor that can influence the choice or steps. of Action type. Some Action types may be more complex Table 2 presents the number of Actions that migrated and time-consuming to maintain than others, which may away from and towards an Action type as well as their prompt developers to switch to another type of Action relative proportion. We observe that composite Actions that is easier to maintain. Finally, integration with other represent the majority of the targets of a migration, retools or services may also be a reason for changing the garding the source type, composite Actions attracted 45.3% Action type. If an Action needs to work with another (73 out of 161) of all migrations away from Docker Actions tool or service that works better with a diferent type and 57.4% (31 out of 54) of all migrations away from Javaof Action, developers may choose to switch to a more Script Actions. Additionally, we note that 15.8% of the appropriate Action type. Docker Actions migrated to either composite or JavaScript Actions. composite Docker JS
Goal G2 aims to characterise the dependency of the three Action types by analysing the GitHub repositories in which these Actions are being developed. Given that diferent Action types have diferent ways to depend on reusable components, we divide this goal into three research questions, one per Action type. 2.1 What are the dependency manifest file for JavaScript projects. The dependencies characteristics of composite Actions? part of this manifest lists the name and version number of the required packages.
Composite Actions represent 11.1% of all Actions (i.e., JavaScript packages and projects are known to be ex312 out of 2,817). Their YAML files define a set of steps posed to security vulnerabilities and other issues coming that combine shell commands and reusable Actions. We from their dependencies [25, 21]. Therefore, we aim to extracted all steps from the composite Actions, distinguish- quantify to which extent JavaScript Actions are relying ing between shell commands (through the run: keyword) on npm packages, as a preliminary step towards quantiand used Actions (through the uses: keyword). The used fying their exposure to security vulnerabilities and the Actions were considered as the dependencies of the com- impact of these vulnerabilities on the GitHub Actions posite Actions. ecosystem.
We found that 73% of the composite Actions (227 out of For each JavaScript Action, we extracted its list of de312) are exclusively composed of 1 to 12 steps executing pendencies from the package.json file of its latest release. shell commands. The remaining 27% (i.e., 85 out of 312) This allowed us to get a list of all direct dependencies combine both shell commands and used Actions. For these Actions have. To obtain their indirect dependencies, those composite Actions that use (i.e., depend on) other we applied the npm-remote-ls 4 command-line tool on Actions, we identified these dependencies and their type. each of its direct dependencies. This tool produces a list If the dependencies were again composite Actions, we of all the packages that are required, taking into account iteratively extracted all steps from their YAML file to ob- the dependency constraints, and also including those that tain a complete list of indirect dependencies of composite are transitively required. For 1,364 out of 1,485 JavaScript Actions. Actions in our dataset, we collected a total of 90,370 (di
Table 3 reports on the dependency depth for composite rect and indirect) dependencies in this way, accounting Actions, the proportion of composite Actions at each depth for 4,309 distinct required packages. The remaining 121 and the types of Actions that composite Actions depend JavaScript Actions do not have a package.json file or the on. The first line, for instance, indicates that there are 85 corresponding package did not available on npm regcomposite Actions (accounting for 27% of the composite istry. Therefore, they are not considered in the following Actions) that have dependencies at depth 1 (i.e., the first analysis. level of dependency nesting). There are 177 dependencies required by these 85 Actions, comprising 23 composite Actions, 5 Docker Actions, and 149 JavaScript Actions. In TDaebpleend4ency characteristics of JavaScript Actions. terms of indirect dependencies, we discovered that 7.4% of composite Actions demonstrated such dependencies, JS Actions # dependencies relying indirectly on only 25 composite Actions. depth # % median mean max Table 4 reports on the depth and proportion of JavaScript Actions that depend on npm packages in the cor2.2 What are the dependency responding depth, along with the median, mean, and characteristics of JavaScript Actions? maximum number of dependencies associated with the JavaScript Actions are written in JavaScript and may de- depth of the dependency trees. For instance, the fifth row pend on npm packages. Developers usually define these shows that among the JavaScript Actions in our dataset, included dependencies in the package.json file, the usual 975 of them (representing 71.5% of the JavaScript Actions) several Linux distributions, each of them having its own have indirect dependencies at a depth of 5, with a median package manager and requiring a diferent methodology and mean value of 4 and 10.8 npm packages, respectively. to identify the packages that are part of the image. This
Our analysis also shows that the median and mean research question therefore focuses on the images used number of direct dependencies for JavaScript Actions in Docker Actions, keeping for future work the identifiwere 4 and 4.3, respectively. Furthermore, we discov- cation of the packages that are actually used within the ered that 99% of the JavaScript Actions contained indirect Docker containers. dependencies, a significant proportion of these depen- We found that the YAML files of 156 out of 1,020 (i.e., dencies being deeply nested. Specifically, we observed 15.3%) of the Docker Actions specify the URL of the Docker that up to 64% of the JavaScript Actions included indirect image directly. The remaining 864 Docker Actions refer dependencies at a depth of 6. The JavaScript Actions are to a Dockerfile instead. We extracted this Dockerfile for considerably exposed to potential issues associated with 842 out of the 864 cases. For the remaining 22 Actions, their dependencies, and since these dependencies may we could not find the corresponding Dockerfile on their be deeply nested, it could become increasingly challeng- repositories. ing to identify and resolve such issues. This situation For each Dockerfile , we extracted the URL of the base is consistent with the observations made for JavaScript image, i.e., the value of the FROM field of the Dockerfile . projects [25, 21]. We parsed this field and extracted the name of the image
In addition, among the npm packages specified as de- and the registry where the image can be found. We did pendencies for JavaScript Actions, @actions/core was the the same for the Actions that directly specify the URL of most frequent npm depencency and 97% of JavaScript Ac- the Docker image in their YAML file. tions in our dataset were dependent on that. The top five Table 5 lists the Docker registries that are the most most frequently occurring npm packages are @actions/- frequently used by Dockerfile s and action.y(a)ml files. We core, @actions/github, @actions/exec, @actions/tool-cache observe that Docker Hub is the most frequently used and @actions/io. The @actions/ namespace corresponds Docker registry. It is used in 95% of the extracted Dockto packages distributed by GitHub to ease the develop- erfile s and in 57.7% of the Action YAML files. This is ment and maintenance of JavaScript Actions, explaining expected, since Docker Hub is the default registry that why these packages are frequently found as dependen- is used for retrieving Docker images when no registry cies for JavaScript Actions. For comparison, the most is explicitly specified. Moreover, Docker Hub is one of required packages that do not belong to this namespace the largest and most active Docker registries. Additionare semver (used by 9.7% of JavaScript Actions), axios ally, during our investigation, we discovered that GitHub (used by 7.8% of JavaScript Actions) and node-fetch (used Container Registry (ghcr.io) ranks as the second most by 5.6% of JavaScript Actions). commonly used Docker registry, used in 36.5% of the Action YAML files. Other registries being reported are the Microsoft Container Registry (MCR), docker.io and Google Container Registry. 2.3 What are the dependency characteristics of Docker Actions? Docker Actions realise their tasks through the execution Table 5 of a Docker container. To define a Docker Action, one Docker image registries used by Docker Actions. has to specify, in the action.y(a)ml file at the root of the repository, either the URL to a Docker image (e.g., from container registry Do#ckerfil%e a#ction.y(a)m%l Docker Hub) or the filepath to a Dockerfile . A Dockerfile is a file specifying the configuration of the container to GitHub ContainDeorcRkeergiHsturyb 80106 951..09 9507 3567..57 be used for the Action. Among other, this file specifies Microsoft Container Registry 15 1.8 0 0.0 the base image that should be used to create the Docker docker.io 3 0.3 3 1.9 container, as well as the various commands that should Google Container Registry 3 0.3 1 0.6 be executed by this container. other registries 5 0.7 5 3.3
Docker images are usually based on Linux distributions and contain several pre-installed packages that are re- We also quantified the base images specified in the quired during the execution of the corresponding Docker Dockerfile . Table 6 reports the top ten most commonly containers. As such, we consider these images and their used images by Docker Actions along with the proportion packages as dependencies for the Docker Actions that of Docker Actions that used them. The numbers suggest use them. However, getting the complete list of pack- that the predominant referenced images are mostly assoages that are required that way by Docker Actions is not ciated with both programming languages (e.g., python, straightforward. As we will see, Docker Actions use a node, golang, etc.) and operating systems (e.g., alpine, large number of images and these images are based on ubuntu, etc.).
Previous research [36, 37, 38] has primarily emphasised the security of system packages inside Docker images, disregarding third-party packages. Considering the growing prevalence of these packages, it is crucial to examine maintainability issues of such third-party packages and the impact of their use in the GitHub Actions ecosystems. The current analysis represents an initial step in studying the dependencies of images used by Docker Actions. As future work, we intend to perform an in-depth study of such Docker images, Dockerfiles and their corresponding commands, with the aim to detect the dependencies of Docker Actions. Investigating the package dependencies within Docker images enables practitioners to enhance the provisioning and deployment of Docker Actions. It can address potential concerns related to maintainability while also promoting the stability of the workflow for users of Docker Actions.
et al. [39] to address the primary threats to validity.
Threats to external validity relate to whether the results can be applied or extended beyond the specific scope of this study. One such threat was our choice to focus the analysis solely on Actions distributed through the GitHub Marketplace. Both the study conducted by Saroar and Nayebi [15] and our own observations highlighted the existence of numerous Actions developed on GitHub repositories that were not published on the GitHub Marketplace. Another threat to external validity relates to ifltering the Actions with at least 10 stars, in order to exclude personal or experimental Actions [35]. Moreover, we exclude around 100 Actions with more than 10 stars that appear in the GitHub Marketplace but produce an error when attempting to load detailed information for them, such as c-documentation-generator5. This implies that our findings may not be universally applicable to all Actions, as filtered and unpublished Marketplace Actions might possess characteristics that difer from those that we analysed.
Threats to construct validity discuss the connection between the theoretical basis of the experiment and the observed results. Our research utilises quantitative observations and rigorous methods in software repository mining, minimizing the impact of threats to construct validity.
Threats to internal validity address the choices and internal factors within the study that have the potential to influence the observations we have made. A potential threat to internal validity arises from the method we used to identify releases of Actions. As described in Section 4, we utilised the "releases" feature of the GitHub API to determine when Actions had been updated. This introduces a potential limitation, since not all Actions depend on the release management system of Github; some may distribute through other mechanisms such as tags, git commits, or branches. However, there is currently no accurate method to identify which git tags and branches correspond to releases, except by using GitHub’s release management system.
Threats to conclusion validity concern the extent to which the conclusions drawn from our analysis are reasonable. Since our conclusions mostly state quantitative observations, it is unlikely that they will be influenced by such threats.
come the de facto CI/CD automation service for GitHub repositories. This preliminary quantitative study aimed to understand the ecosystem of GitHub Actions and its dependency on external, third-party components such as Docker images and npm packages. Through an analysis of the GitHub development repositories of 2,817 Actions distributed through the GitHub Marketplace, we aimed to explore the characteristics of their dependencies. To do so, we distinguished between three types of Actions: those that are developed using JavaScript, those that are developed using Docker, and the so-called composite Actions.
We observed that, while most Actions are developed using JavaScript, a significant proportion of Actions are based on Docker. Moreover, after GitHub decided to introduce the mechanism of composite Actions, such Actions seem to be slowly replacing Docker Actions. Considering 5https://github.com/marketplace?type=actions&query= c-documentation-generator+
Scientifique - FNRS under grant numbers PDR T.0149.22 and F.4515.23. the fact that composite Actions can depend on other Actions, this might lead to more (transitive) dependencies in the GitHub Actions ecosystem.
JavaScript Actions have many dependencies towards npm packages, leading to a huge number of transitive dependencies that are deeply nested in the dependency tree. This makes JavaScript Actions considerably exposed to potential maintainability issues associated with their dependencies. It could become increasingly challenging to identify and resolve such issues, similar to what has been observed for JavaScript projects and packages [21, 25, 26, 40, 41].
We observed that Docker Hub is the most popular container registry for Docker Actions and the predominant used images by Docker Actions are associated with programming languages and operating systems. The GitHub container registry is another container registry that is commonly used by Action YAML files.
As future work, we intend to assess the impact of maintainability issues such as dependency management, security vulnerabilities, compatibility issues, outdated or obsolete components on the GitHub Action ecosystem due to its cross-ecosystem dependencies to Docker images and npm packages. ware Engineering 24 (2019) 381–416. doi:10.1007/ age dependency networks, IEEE Transactions on s10664-017-9589-y. Software Engineering (2021). doi:10.1109/TSE. [17] W. B. Frakes, K. C. Kang, Software reuse research: 2021.3112204.
status and future, IEEE Transactions on Software [29] C. Bogart, C. Kästner, J. Herbsleb, F. Thung, When Engineering 31 (2005) 529–536. doi:10.1109/TSE. and how to make breaking changes: Policies and 2005.85. practices in 18 open source software ecosystems, [18] A. Decan, T. Mens, M. Claes, An empirical compari- ACM Transactions on Software Engineering and son of dependency issues in OSS packaging ecosys- Methodology (TOSEM) 30 (2021) 1–56. doi:10. tems, in: International Conference on Software 1145/3447245.
Analysis, Evolution and Reengineering (SANER), [30] A. Decan, T. Mens, E. Constantinou, On the IEEE, 2017. doi:10.1109/SANER.2017.7884604. evolution of technical lag in the npm package [19] R. Abdalkareem, O. Nourry, S. Wehaibi, S. Mujahid, dependency network, in: International ConE. Shihab, Why do developers use trivial pack- ference on Software Maintenance and Evolution ages? An empirical case study on npm, in: Joint (ICSME), IEEE, 2018, pp. 404–414. doi:10.1109/ Meeting on Foundations of Software Engineering ICSME.2018.00050. (FSE), 2017, pp. 385–395. doi:10.1145/3106237. [31] A. Zerouali, T. Mens, J. Gonzalez-Barahona, A. De3106267. can, E. Constantinou, G. Robles, A formal frame[20] C. Soto-Valero, N. Harrand, M. Monperrus, work for measuring technical lag in component B. Baudry, A comprehensive study of bloated repositories—and its application to npm, Journal dependencies in the Maven ecosystem, Em- of Software: Evolution and Process 31 (2019) e2157. pirical Software Engineering 26 (2021) 45. URL: doi:10.1002/smr.2157. https://doi.org/10.1007/s10664-020-09914-8. doi:10. [32] F. Cogo, G. Oliva, A. E. Hassan, Deprecation of 1007/s10664-020-09914-8. packages and releases in software ecosystems: A [21] A. Decan, T. Mens, E. Constantinou, On the impact case study on npm, IEEE Transactions on Softof security vulnerabilities in the npm package de- ware Engineering (2021). doi:10.1109/TSE.2021. pendency network, in: International Conference 3055123. on Mining Software Repositories (MSR), 2018, pp. [33] S. Mirhosseini, C. Parnin, Can automated pull 181–191. doi:10.1145/3196398.3196401. requests encourage software developers to up[22] A. Zerouali, T. Mens, A. Decan, C. De Roover, On grade out-of-date dependencies?, in: International the impact of security vulnerabilities in the npm and Conference on Automated Software Engineering RubyGems dependency networks, Empirical Soft- (ASE), 2017, pp. 84–94. doi:10.1109/ASE.2017. ware Engineering 27 (2022) 1–45. doi:10.1007/ 8115621.
s10664-022-10154-1. [34] J. Cox, E. Bouwers, M. C. J. D. van Eekelen, J. Visser, [23] M. Alfadel, D. E. Costa, E. Shihab, E. Shihab, Empir- Measuring dependency freshness in software sysical analysis of security vulnerabilities in Python tems, in: International Conference on Software Enpackages, in: International Conference on Software gineering (ICSE), IEEE, 2015, pp. 109–118. doi:10. Analysis, Evolution and Reengineering (SANER), 1109/ICSE.2015.140.
2021. doi:10.1109/saner50967.2021.00048. [35] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, [24] H. H. Thompson, Why security testing is hard, D. M. German, D. Damian, The promises and perils
IEEE Secur. Priv. 1 (2003) 83–86. of mining GitHub, in: International Conference on [25] T. Lauinger, A. Chaabane, C. B. Wilson, Thou shalt Mining Software Repositories (MSR), ACM, 2014, not depend on me, Communications of the ACM pp. 92–101. doi:10.1145/2597073.2597074. 61 (2018) 41–47. doi:10.1145/3190562. [36] R. Shu, X. Gu, W. Enck, A study of security vulner[26] C. Liu, S. Chen, L. Fan, B. Chen, Y. Liu, X. Peng, abilities on docker hub, Proceedings of the Seventh Demystifying the vulnerability propagation and ACM on Conference on Data and Application Seits evolution via dependency trees in the npm curity and Privacy (2017). doi:10.1145/3029806. ecosystem, International Conference on Software 3029832.
Engineering (ICSE) (2022) 672–684. doi:10.1145/ [37] J. Gummaraju, T. Desikan, Y. Turner, Over 30% of 3510003.3510142. oficial images in docker hub contain high priority [27] A. Decan, T. Mens, What do package dependencies security vulnerabilities, Technical Report (2015). tell us about semantic versioning?, IEEE Transac- [38] A. Zerouali, T. Mens, G. Robles, J. M. Gonzaleztions on Software Engineering 47 (2019) 1226–1240. Barahona, On the relation between outdated doi:10.1109/TSE.2019.2918315. docker containers, severity vulnerabilities, and [28] A. Decan, T. Mens, A. Zerouali, C. De Roover, Back bugs, in: 2019 IEEE 26th International Conference to the past–analysing backporting practices in pack- on Software Analysis, Evolution and Reengineering (SANER), 2019, pp. 491–501. doi:10.1109/SANER.
2019.8668013. [39] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson,
B. Regnell, A. Wesslén, Experimentation in Software Engineering, Springer, 2012. [40] I. S. Makari, A. Zerouali, C. D. Roover, Prevalence and evolution of license violations in npm and rubygems dependency networks, in: International Conference on Software Reuse, 2022. doi:10.
1007/978-3-031-08129-3_6. [41] A. Zerouali, E. Constantinou, T. Mens, G. Robles,
J. M. Gonzalez-Barahona, An empirical analysis of technical lag in npm package dependencies, in: ICSR, 2018. doi:10.1007/978-3-319-90421-4_ 6.