CI/CD practices play a significant role in collaborative software development. The GitHub social coding platform introduced GitHub Actions as a way to automate diferent aspects of software production such as testing, building, quality checking, dependency and security management. We report on preliminary findings of a quantitative analysis on how GitHub Actions workflows are being changed over time. The study is based on a dataset of 22,733 GitHub repositories containing 4 million weekly snapshots of workflow lfies from November 2019 to September 2022. First, we analyse the coarse-grained changes being made to workflows, including when repositories start using them, when they are being added, modified, renamed and removed. Second, we analyse changes made to workflow code, including how many code lines are changed and what types of changes are being made to them. The findings of this quantitative analysis provide preliminary insights on how GitHub Actions workflows are being changed over time, and whether they adhere to the evolution laws of continuing growth and continuing change. It paves the way for studying more evolution laws, as well as more in-depth analyses on the types of changes that CI/CD workflows are subject to as well as the reasons for these changes.
18 months [8]. Decan et al. [9] reported that, out of 68K+
GitHub repositories provided in their dataset, 43.9% are
Continuous Integration and Development (CI/CD) prac- utilizing GHA workflows by the end of January 2022.
tices aim to help developers release high-quality software This indicates the widespread adoption of GHA within
products more eficiently and with less efort [
DevOps, CircleCI, AppVeyor, and GitHub Actions) to au- Knowing when, why and how developers modify
worktomate a broad range of software development activities lfow files can be helpful to improve CI/CD practices, to
such as testing, building, quality checking, dependency detect common patterns and mistakes developers do in
and security management [
Focusing on the GitHub social coding platform in par- writing and maintaining workflows. ticular, which is the largest collaborative software devel- As preliminary steps towards such a comprehension, opment platform used by software developers, GitHub this article aims to characterise the changes made to these Actions (GHA) was introduced as a new CI/CD solution workflow files over their lifetime. To do so, we study in November 2019. It has quickly gained popularity, re- two main research goals, based on an extracted dataset placing Travis as the dominant CI/CD tool in less than of 22,733 repositories accounting for 4,127,760 weekly snapshots of workflow files over a 34-month period from November 2019 to September 2022.
Goal G1 aims to quantify the coarse-grained changes being made to GHA workflows, and is broken down into three specific research questions: 1.1 When do repositories start using GHA? As the most primitive type of change, we investigate how long it takes for repositories to start using GHA. 1.2 Which types of coarse-grained changes are
Goal G2 focuses on quantifying the fine-grained changes to workflow files, by analysing the changes that are made in their contents. In this article, we restrict ourselves to studying line-based changes through the two following research questions: 2.1 Which types of line-based changes are worklfow files subject to? We examine the addition, modification and removal of lines in workflow files. 2.2 When do diferent types of line-based changes occur? For the three above types of fine-grained worklfow changes, we analyse when they occur over time. workflows subject to? We analyse the four possible Durieux et al. [12] compiled a dataset of over 35M+ types of coarse-grained changes that can be made to Travis jobs triggered by 272,917 projects. Their findings workflows: adding, renaming or removing a workflow indicate that out of the 709K+ commits that changed ifle, as well as modifying the contents of the workflow a Travis configuration file, the majority are related to ifle. debugging the configuration file. They also suggest the 1.3 When do the diferent types of coarse-grained necessity for a more in-depth analysis of the nature of changes occur? For all four types of coarse-grained the changes made to those commits. workflow changes, we analyse when these changes oc- Zampetti et al. [13] identified 79 bad CI practices by cur. conducting semi-structured interviews with 13 experts and analyzing over 2,300 Stack Overflow posts, considering posts with four diferent tags: continuous integration, Jenkins, Hudson, and travis-ci. Additionally, Zampetti et al. [14] studied the evolution of changes to Travis configuration pipelines. They found that jobs and steps were the most frequently changed. Furthermore, they observed an increasing trend in the adoption of Docker by projects over time.
2. Related Work significantly impacted the CI/CD landscape for GitHub repositories. Several studies have been conducted to This section explores the existing literature on the evolu- investigate the impact of GHA on the overall usage of tion of CI/CD configuration files before the emergence CI/CD tools. of GitHub Actions (Section 2.1) and the use of GHA as a Golzadeh et al. [8] conducted a longitudinal quantitaCI/CD tool (Section 2.2). To the best of our knowledge, tive study on the adoption of CI/CD tools in 91K+ GitHub the current article is the first large-scale study investigat- repositories related to npm packages. The study revealed ing the evolution of GHA workflow file changes. a growing reliance on CI/CD tools, with more than 50% of repositories utilizing such tools as of May 2021. The 2.1. Evolution of CI/CD configuration authors found that GHA and Travis were the dominant files CI/CD tools, being used by 90% of the considered repositories that relied on a CI/CD tool. They also observed Even before introduction of GHA, usage of CI/CD tools that GHA replaced Travis as dominant CI/CD tool in in software projects was well established and most well just 18 months after its introduction as a consequence of known ones of them (especially Travis) have been using many repositories migrating from Travis to GHA. YAML based configuration files. In this section, we go Kinsman et al. [15] analysed the impact of adopting through studies related to the evolution of the configura- GHA in 3,190 repositories. They found that the adoption tion file in those CI/CD tools. of GHA resulted in an increase in the number of rejected
Gallaba and McIntosh [10] studied the usage and mis- pull requests and a decrease in the number of commits in use of features in Travis configuration files based on a merged pull requests. Based on a manual inspection of dataset of 9,312 GitHub repositories. They found that 209 GHA-related issues, they concluded that developers job processing nodes were the most frequently modified, had a generally positive perception of GHA. These findindicating that most of the usage of Travis was for con- ings were confirmed by Chen et al. [ 16], who conducted tinuous integration rather than continuous deployment. a replication study on 6,246 repositories. They also developed and evaluated a tool for identifying Decan et al. [9] conducted an analysis of the use of anti-patterns in Travis configuration files, and a second GHA in almost 70K+ GitHub repositories to gain a deeper tool for automatical removal of those anti-patterns. understanding of the GHA ecosystem. They found that
In a similar vein, Vassallo et al. [11] developed a tool to 43.9% of the repositories used GHA workflows and
charidentify anti-patterns in Java projects relying on Travis. acterized these repositories and their workflow contents
To determine the most critical anti-patterns in Travis log in terms of the usage of jobs, steps, and reusable Actions.
ifles, they relied on Duvall’s work [
Valenzuela-Toledo and Bergel [17] conducted a prelim- tory (in this case: master). A common change consists of inary study to examine the usage and maintenance of adding new steps (e.g., step ‘Publish Test Report’ on lines GHA workflows in ten popular GitHub repositories. They 17-19 on the right) or adding more lines to existing steps analyzed 222 commits to propose an initial taxonomy of (e.g., lines 12 and 14 on the right that add extra arguments workflow modifications. for the Java setup; line 26 that added an extra argument
Rostami Mazrae et al. [18] qualitatively investigated for the JaCoCo badge generator). Line 5 on the right is the reasons behind the adoption of CI/CD tools in soft- an example of a change to a job, by adding an extra name ware projects, the co-usage of multiple CI/CD tools, and to it. Another common change is modifying the contents the migration from one CI/CD tool to another one. For of existing lines, for example to update the version of GitHub projects more specifically, they investigated the the Action being used (e.g., from actions/checkout@v2 factors that led to GHA becoming the dominant CI/CD to actions/checkout@v3, from actions/setup-java@v1 to tool. The majority of reported migrations towards GHA actions/setup-java@v3, and so on), to edit the name of a were due to its strong integration with GitHub, its ease step (line 31 on the right), to modify the Java version to of use, and its large marketplace of Actions. use for the build (line 13 on the right), and so on.
Saroar et al. [19] surveyed 90 Action producers and This example shows the various possible changes in users to understand the motivations and best practices workflow files and justifies the need for studies on when, in using, developing, and debugging Actions, and the why, and how developers modify workflow files. Such challenges associated with these tasks. They report that studies are likely to improve CI/CD practices, for examusers prefer Actions with verified producers and more ple, by identifying common patterns and issues during stars when choosing between similar Actions, and often workflow modifications and by providing tools to assist switch to an alternative Action when facing bugs or a lack developers in writing and maintaining workflows. of documentation. Moreover, they report that more than half of the Action producers consider the composition of YAML files challenging and error-prone and would 4. Data extraction mainly check question and answer forums to fix issues with these YAML files. To study the evolution of workflow files, a large dataset of
Wessel et al. [20] suggested to consider studying GitHub GitHub repositories relying on GHA is required. We used Actions as a software ecosystem that faces similar chal- the SEART [23] GitHub search engine1to select reposilenges as traditional software library ecosystems [21, 22]. tories. To mitigate the usual threats related to software repository mining [24], we excluded repositories that are used only for experimental or personal purposes, or 3. Motivating example that exhibit minimal evidence of software development activity. To do so, we selected repositories created before 2022 that were still active in 2022, had at least 100 stars and 100 commits, and were not forks. Considering these constraints, the final list of repositories included 62,673 instances.
On September 2022, we locally cloned these repositories to identify the presence of GHA workflow files in the .github/workflows directory of their default branch (as reported by the GitHub API) and found 22,733 repositories satisfying this criterion. Since our goal is to study the evolution of the workflow files in these repositories, we relied on a combination of the git rev-list and git checkout CLI tools to materialize the content of each workflow file in each repository for every Monday between November 2019 (the oficial release date of GHA) and September 2022, accounting for 148 time points. By considering weekly snapshots instead of all the commits that modified the workflow files, we mitigate the usual threats related to commits performed on parallel branches that are eventually merged [25]. This
workflow files must be created and stored within the .github/workflows folder. As with any other software development component, workflow files evolve over time to better serve their purposes.
Figure 1 presents a visual dif of some changes made to a workflow file that automates the building, testing and code coverage analysis of some Java project. On the left is the old version, with lines highlighted in red representing removed or changed lines, and the dark red parts highlighting the parts of the line that were changed. On the right is the new version, with lines highlighted in green representing new or changed lines, and the dark green parts highlighting the changes that were made w.r.t. the previous version.
One can observe that changes may occur in diferent locations for diferent reasons. For instance, line 2 illustrates a change related to the events that trigger the workflow. This particular change can be considered a behaviour-preserving refactoring and simplification of
resulted in a dataset of 4,127,760 workflow file snapshots (of which 271,422 are unique) in 22,733 GitHub repositories.
Figure 2 shows the evolution of the number of repositories and workflow files through time. We observe that both numbers are continuously increasing through time, indicating that more and more repositories are making use of GHA and more and more workflow files are created in these repositories. At the end of the observation period, there are 65,067 workflow files spread in 22,733 repositories. The figure also reveals a slight disturbance in the evolution of both numbers in November and December 2020. This coincides with restrictions imposed by Travis CI on its free plan for public repositories, which caused many repositories to switch from Travis CI to GHA during these two months, as already observed by Golzadeh et al. [8]. 5. Goal G1: Coarse-grained
changes in GHA workflows Research goal 1 aims to analyze and measure the coarsegrained changes that occur to GHA workflows in GitHub repositories. To accomplish this goal, we will answer three research questions. 50000 40000 30000 20000 10000 0 repositories workflows
ries, we analyzed the time it took for repositories to start using it. Previous studies [8, 18] have reported that GHA has become the dominant CI/CD tool on GitHub. This research question aims to understand the time it takes for repositories to adopt GHA as their CI/CD tool.
We distinguish between the repositories that already existed when GHA was introduced on GitHub and those that were created after. Indeed, repositories that were created before the introduction of GHA were likely to use another CI/CD tool before migrating to GHA and such a migration does not come for free. In their cases, we are interested in the time they took to adopt GHA since its oficial release in November 2019. On the other hand, repositories that were created after the introduction of GHA are more likely to adopt it as their CI/CD tool because of its deep integration into GitHub. In their cases, we are interested in the time they took to adopt GHA since these repositories were created.
We therefore divided the repositories in our dataset into two categories: those that already existed before the public release of GHA in November 2019, accounting for 18,805 repositories; and the remaining 3,928 that were created after.
The time to adopt GHA depends on whether the repository was created before or after GHA’s oficial release. A majority of the repositories that were created after GHA adopted it within a few months after their creation. On the other hand, it took more than one year since GHA’s release for most of the older repositories to adopt GHA. 1.2 Which types of coarse-grained changes are workflows subject to? 1.0 This research question aims to identify the kind of coarse0.9 grained changes workflow files are subject to. We disiton 0.8 tinguish between the four following types of changes: roop irsoe00..67 addition, modification , renaming, or removal. To keep litrvaeup iftrsooep000...453 ifle a unique identifier that is preserved through renamtrack of these changes, we attributed to each workflow cum 00..12 repositoabrfeitefeosrrecGrHGeAaHtAed icnagns.cIot-ioscwcuorrt(he.mg.,enat ifiloencinang tbheatresonmame eodf tahtetsheechsaanmgees 0.0 time than its content is modified) while some changes 0 5 de1l0ay to int1ro5duce G2H0A (in mo2n5ths) 30 35 are causally dependent (e.g., a workflow file can only be removed after having been added previously).
Figure 3: Cumulative proportion of repositories in function We detect the four diferent types of changes as folof the time (in months) to start using GHA. lows. The addition of a workflow file is detected when the workflow file is seen for the first time in a snapshot. Sim
Fig. 3 shows the cumulative proportion of repositories ilarly, the removal of a workflow file is detected when the in function of the time they took to start using GHA. file is no longer visible in a snapshot. The modification of Its worth recalling that our dataset only contains repos- a workflow file is detected in a snapshot by comparing itories that are still using GHA in the latest considered its content in the current snapshot with its content in snapshot, explaining why the proportions reach 100%. the previous snapshot. These modifications are detected
Focusing first on the repositories that were created by comparing the SHA-256 hashes in consecutive workafter GHA’s public release (orange line in Fig. 3), we ob- flows. Finally, the renaming of a workflow file is detected serve that it takes less than 3 months for 50% of them to based on the following heuristic. The heuristic detects a adopt GHA after their creation, as indicated by the left- renaming from to when is removed at the same most dotted line. After only 10 months, this proportion time that is added. If and have exactly the same reaches 75%. The adoption rate for the repositories that content (i.e., they have the same SHA-256 hash), we conalready existed when GHA was released (blue line) is sider that was renamed to . If and difer in their much lower: only 15% of these repositories adopted GHA content, we check whether there is no other workflow after 3 months and it required 10 months to reach 50% ifle that was added or removed at the same time. If of the repositories (as indicated by the rightmost dotted there is no such , then we consider that was renamed line), and even 21 months to reach 75%. to . By doing so, we ensure we are not exposed to false
This supports our hypothesis that it takes more time positives even if this implies we may miss renamings, for repositories already in place to adopt GHA, likely e.g., in the case multiple workflow files are renamed at because they were already using another CI/CD tool, the same time. implying a longer delay to start using GHA due to the For each repository and each pair of consecutive snaptechnical or organizational dificulties that may come shots, we relied on the above approaches to detect changes. with a migration to a new CI/CD solution, or simply due The most frequent workflow change type is modification , to the lack of need to carry out such a migration [8, 18]. accounting for 73% of all changes. The second most freOn the other hand, adopting GHA in newer repositories quent change type is addition, accounting for 22.8% of is much easier thanks to the tight integration of GHA all changes. Removal is less common, accounting for with GitHub, its ease of use, its low learning curve, and less than 3.9% of changes. Renaming is the least comthe ease of setting up new workflow files from scratch mon change type, accounting for 0.1% of all changes over based on suggested configurable templates [19, 18]. entire considered period.
addition modification removal renaming
Since a repository may have many workflow files be- since GHA was introduced in each repository. We obing changed at the same time, we also computed for each serve that the proportion of repositories exhibiting a snapshot the proportion of repositories exhibiting the ad- change, regardless of the change type, is higher during dition, removal, modification and renaming of a workflow the first weeks, and that this proportion quickly decreases ifle. Figure 4 shows the evolution of these proportions. through time to reach a quite stable value. For instance, We observe that the most frequent change is modification , more than 15% of the repositories made changes to their exhibited in around 9% of the repositories at the begin- workflow files during the first week, but this proportion ning of the observation period and slowly decreasing to decreases to around 6% after six weeks only. Similarly, around 6% of the repositories at the end of the observa- the proportion of repositories adding workflow files detion period. Unsurprisingly, additions and removals are creased from around 5% in the first week to approxiless frequently observed in repositories, ranging from mately 1.5% after six weeks. 2.4% to 0.7% and from 1.5% to 0.2% of the repositories, re- The slight variations that can be observed starting from spectively. Finally, renamings are barely never observed week 100 is a consequence of the much lower number of in the considered repositories. repositories that have been using GHA for 100 weeks or more. For instance, while we have 22K repositories for Each week, on average, 7.2% of the repositories mod- the first week, we only have 4.8K repositories at week ify a workflow file, while 1.4% of them add a new 100, 2.4K at week 120, and 574 at week 140. workflow and 0.5% remove a workflow. These observations suggest that workflow files follow Lehman’s evolution laws [26] of continuing change (workflows are regularly modified through time) and continuing growth (workflows are being added more often 1.3 When do diferent types of than they are removed). Although it is not surprising that coarse-grained changes occur? workflow files are regularly modified in order to integrate new pipelines or new functionalities, we postulate that Previous research question reported on the various types part of the observed modifications are the consequence of of changes that occur in workflow files. With the current the dificulty to debug, test and validate workflows. This research question, we aim to understand when those challenge was already identified by Saroar et al. [ 19] and changes occur with respect to the adoption of GHA in in a qualitative study conducted by Rostami et al. [18]: each repository. We posit that the first few weeks after “You will see that you do a lot of typos and try to run the introducing GHA, workflow maintainers are likely to CI/CD 20 times until it works once. You copy-paste some make many changes until the workflows reach a stable examples from the Internet, you adapt it, but you forget to state and that, afterwards, only occasional changes are like there’s a lot of details. It’s often YAML files that are needed further. In order to verify this hypothesis, for each really prone to mistakes. So you make [lots of] commits change detected in previous RQ, we computed the time until you get to the result you want to have. And there’s between the introduction of GHA in the corresponding no way to pre-test it on your local machine. So you just repository and the date of the change to the workflow commit, push, wait for the build to run, and then look at ifle. the results.”
ifles within the first weeks after having adopted GHA.
Nevertheless, we observe that each week, around 6% of the repositories modify a workflow file. This confirms that workflows are subject to the laws of continuing change and continuing growth. 6. Goal G2: Fine-grained changes
in GHA workflows The second research goal G2 focuses on analysing finegrained changes in workflow files at a line-based level. 2.1 Which types of line-based changes are workflow files subject to? added
As a first step towards reaching our goal of studying the ifne-grained changes in GHA workflows, we aim to iden- 3.45% + 3.06% + 6.09% + 13.63%) involve removing lines. tify how frequently lines are added, removed and modi- It is noteworthy that removing lines alone is infrequent. ifed in workflow files. To do so, we relied on the CLOC command-line tool [27]. CLOC is a tool that counts the Nearly all the changes made to workflow files innumber of lines in files. It has an option to compare volve lines of code accounting for almost 9 out of two files, and reports on the number of lines that were 10 lines added, removed or modified. Modifying and added, removed, modified and untouched between the two adding lines are the most frequent operations made ifles, distinguishing between lines of code, blank lines and to workflow files. comments.
We applied CLOC on all the workflow files that were modified, comparing the content of each workflow file with the content of the previous version of this workflow 2.2 When do diferent types of ifle. We found that the very large majority of the changes line-based changes occur? are related to lines of code. For instance, 87.8% of the lines We already observed in 1.3 that, while repositories that were either added, removed or modified are lines are more likely to change their workflow files within the of code. A further examination of the modifications of very first weeks after having adopted GHA, changes are the workflow files indicates that 69% of the modifications nevertheless observed during the whole lifetime of these made to a workflow exclusively involve lines of code, 15% ifles. With 2.2, we aim to gain a better understanding involve code and blanks, 7% involve code and comments, of the type of changes that are made to lines of code in and 6.6% involve all types of lines (i.e., code, comment workflow files, hypothesizing that many lines of code will and blank lines). Overall, that implies that 97.6% of the be added to the workflow files during their first weeks modifications made to workflow files involve lines of to add new functionalities until a stable point is reached, code. and then lines will be mostly modified for maintenance
We also looked at how frequently lines are added, re- purposes. moved or modified. Figure 6 shows a Venn diagram re- As a first step, we start by computing the proportion porting on the proportion of workflow modifications in of lines of code that are touched (i.e., lines that are added, which lines were added, modified or removed (or any removed or modified) during workflow changes. Figure 7 combination of those). We observe that 40.11% of the shows the evolution of this proportion in function of changes consist of line modifications only, and lines are the time elapsed since GHA was adapted by the corremodified in 78.69% (40.11% + 18.86% + 6.09% + 13.63%) of sponding repositories. The figure shows the median and the changes. Adding lines is the second most frequent mean values, as well as the 25th and 75th percentiles, repchange observed in workflow files, especially in combi- resented by the shaded light blue area. As explained in nation with modifying lines. For instance, half of the 1.3, the higher variation on the rightmost part of the changes (50.13% = 14.58% + 18.86% + 3.06% + 13.63%) in- figure are due to the lower number of repositories that volves adding lines, while “only” 26.23% of the changes (= have been using GHA for more than 100 weeks.
We observe that both the median and mean values exhibit a gradual decrease during the first year. The diference between the median and mean values indicates that the distribution exhibits a positive skew. Focusing on the mean value, the decrease in the number of lines of code touched is particularly visible in the first weeks, going from an average of 13.6% during the first six weeks to an average of 10.2% for the next year. This indicates that more changes are applied to the lines of workflow ifles in the early phase of the workflows’ lifetime.
Since a line can be touched either because it is added, removed or modified, the two following analyses focus on the evolution of the number of lines that are added and modified. We do not report on the number of lines that are removed given that we observed in 2.1 that removing lines is very infrequent.
0 20
Figure 8 shows the evolution of the distribution of the number of lines of code added in workflow files, in function of the time elapsed since GHA has been adapted by their corresponding repositories. As observed in 2.1, only half of the changes involve adding lines, explaining why the median value is very low (either 0 or 1). Focusing on the mean number of added lines of code, we observe a gradual decrease through time. For instance, the mean number of added lines decreased from an average of 7.4 during the first six weeks to an average of 5.6 during the next year. 10 8 6 4 2 0 median mean 0 20
Internal validity relates to the extent to which the study results are influenced by the experimental treatment or condition being studied [28]. We analyzed the evolution of workflows in software development repositories by studying weekly snapshots.
Although one can study workflows commit by commit, Bird et al. [25] stated that git lacks a mainline and a file can change in parallel in diferent branches, which makes it dificult to track the linear history of a file. Therefore, we chose to analyze snapshots instead of checking com- workflow file changes. These changes are present in over mits directly. 95% of all snapshots studied and are predominantly in
External validity concerns the generalisability of the the form of line modification, addition, or a combination results [28]. We only consider public software reposi- of both. Furthermore, we observed that the proportion of tories with more than 100 stars and commits which are untouched lines generally increases with the workflow’s still under development and maintenance. These criteria lifetime, whereas most line additions occur within the are used to find projects best suited for software evolu- ifrst six weeks. In contrast, code modification occurs tion studies in the case of GHA CI/CD tool. However, steadily throughout the workflow files. we can not generalize these findings on other GitHub This preliminary research showed the prevalence of repositories, including personal webpages. changes in GitHub Actions workflow files. In future work,
Construct validity concerns the relation between we aim to seek evidence for two other laws of software the theory behind the experiment and the observed find- evolution postulated by Lehman [26], namely increasings [29]. To detect the use of workflows in GitHub ing complexity (stating that complexity increases unless repositories, we identified the presence of a YAML file in work is done to maintain or reduce it) and declining qualthe .github/workflows folder. This approach may lead to ity (stating that quality will decline unless the system an overestimation of the presence and correctness of a is rigorously maintained and adapted to operational enYAML file. It can be due to problems in the correspond- vironment changes). We will therefore study to which ing workflow files or simply not being triggered for use. extent we can find evidence of quality-related problems However, we believe that most of these workflows are such as code smells in workflow files, as well as the efindeed used since developers are unlikely to keep GHA fect of preventive changes such as refactorings aimed at workflows without using them or do not solve the issues increasing the quality and reducing the complexity of in the workflow files. workflow files.
Conclusion validity threats concern the degree to which reasonable conclusions have been derived from our analysis [30]. As our results only report quantitative Acknowledgments observations, they are not exposed to such threats.
workflows in collaborative software development by conducting a quantitative analysis on a dataset of 22,733 GitHub repositories containing over 4 million weekly snapshots of workflow files. This study aimed to provide preliminary findings on the changes made to these workflows over time.
As a first goal, we aimed to quantify the coarse-grained changes in GitHub Actions workflows in software projects. Our findings revealed that the majority of repositories created after the introduction of GHA tend to adopt it as their primary CI/CD tool within a few months. Furthermore, we investigated the types of changes observed in workflow files and found that modifications are the most common type of change, followed by additions, with a significant diference between the two. This seems to confirm that Lehman’s software evolution laws of continuing change and continuing growth [26] also hold for workflow files. A notable diference with the evolution of regular source code, however, appears to be that most of the changes made to workflows occur within the first six weeks of adopting GHA in the project.
As a second goal, we aimed to quantify the fine-grained line-based changes in GHA workflow files. Our findings reveal that changes to code lines constitute 87.8% of all
tion de Recherche Concertée financée par le Ministère de la Communauté française - Direction générale de l’Enseignement non obligatoire et de la Recherche scientifique, and by the Fonds de la Recherche Scientifique FNRS under grant number F.4515.23. [6] M. Hilton, T. Tunnell, K. Huang, D. Marinov, D. Dig, GitHub Action workflows, in: International ConferUsage, costs, and benefits of continuous integration ence on Software Analysis, Evolution and Reengiin open-source projects, in: International Confer- neering (SANER), IEEE, 2022, pp. 123–127. ence on Automated Software Engineering (ASE), [18] P. Rostami Mazrae, T. Mens, M. Golzadeh, A. Decan, IEEE, 2016, pp. 426–437. On the usage, co-usage and migration of CI/CD [7] M. Beller, G. Gousios, A. Zaidman, Oops, my tests tools: A qualitative analysis, Empirical Software broke the build: An explorative analysis of Travis Engineering 28 (2023) 52.
CI with GitHub, in: International Conference on [19] S. G. Saroar, M. Nayebi, Developers’ perception Mining Software Repositories (MSR), IEEE, 2017, of GitHub Actions: A survey analysis, in: Internapp. 356–367. tional Conference on Evaluation and Assessment [8] M. Golzadeh, A. Decan, T. Mens, On the rise and fall in Software Engineering (EASE), 2023. of CI services in GitHub, in: International Confer- [20] M. Wessel, T. Mens, A. Decan, P. Rostami Mazrae, ence on Software Analysis, Evolution and Reengi- The GitHub development workflow automation neering (SANER), 2022. ecosystems, in: Software Ecosystems: Tooling and [9] A. Decan, T. Mens, P. R. Mazrae, M. Golzadeh, On Analytics, Springer, 2023.
the use of GitHub Actions in software development [21] A. Decan, T. Mens, P. Grosjean, An empirical comrepositories, in: International Conference on Soft- parison of dependency network evolution in seven ware Maintenance and Evolution (ICSME), IEEE, software packaging ecosystems, Empirical Soft2022. ware Engineering 24 (2019) 381–416. [10] K. Gallaba, S. McIntosh, Use and misuse of contin- [22] A. Decan, T. Mens, E. Constantinou, On the imuous integration features: An empirical study of pact of security vulnerabilities in the npm package projects that (mis) use Travis CI, Transactions on dependency network, in: Proceedings of the 15th Software Engineering 46 (2018) 33–50. international conference on mining software repos[11] C. Vassallo, S. Proksch, H. C. Gall, M. Di Penta, Au- itories, 2018, pp. 181–191.
tomated reporting of anti-patterns and decay in [23] O. Dabic, E. Aghajani, G. Bavota, Sampling projects continuous integration, in: International Confer- in GitHub for MSR studies, in: International Conference on Software Engineering (ICSE), IEEE, 2019, ence on Mining Software Repositories (MSR), IEEE, pp. 105–115. 2021, pp. 560–564. [12] T. Durieux, R. Abreu, M. Monperrus, T. F. Bissyandé, [24] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, L. Cruz, An analysis of 35+ million jobs of Travis CI, D. M. German, D. Damian, An in-depth study of the in: International Conference on Software Mainte- promises and perils of mining GitHub, Empirical nance and Evolution (ICSME), IEEE, 2019, pp. 291– Software Engineering 21 (2016) 2035–2071. 295. [25] C. Bird, P. C. Rigby, E. T. Barr, D. J. Hamilton, D. M. [13] F. Zampetti, C. Vassallo, S. Panichella, G. Canfora, German, P. Devanbu, The promises and perils of H. Gall, M. Di Penta, An empirical characterization mining git, in: International Working Conference of bad practices in continuous integration, Empiri- on Mining Software Repositories (MSR), IEEE, 2009, cal Software Engineering 25 (2020) 1095–1135. pp. 1–10. [14] F. Zampetti, S. Geremia, G. Bavota, M. Di Penta, [26] M. M. Lehman, Laws of software evolution revisCI/CD pipelines evolution and restructuring: A ited, in: European Workshop on Software Process qualitative and quantitative study, in: International Technology (EWPST), Springer, 1996, pp. 108–124. Conference on Software Maintenance and Evolu- [27] A. Danial, CLOC, 2021. doi:10.5281/zenodo. tion (ICSME), IEEE, 2021, pp. 471–482. 7455676. [15] T. Kinsman, M. Wessel, M. A. Gerosa, C. Treude, [28] A. Ampatzoglou, S. Bibi, P. Avgeriou, M. Verbeek, How do software developers use GitHub Actions A. Chatzigeorgiou, Identifying, categorizing and to automate their workflows?, in: International mitigating threats to validity in software engineerConference on Mining Software Repositories (MSR), ing secondary studies, Information and Software IEEE, 2021, pp. 420–431. Technology 106 (2019) 201–230. [16] T. Chen, Y. Zhang, S. Chen, T. Wang, Y. Wu, Let’s [29] P. Ralph, E. Tempero, Construct validity in softsupercharge the workflows: An empirical study of ware engineering research and software metrics, GitHub Actions, in: International Conference on in: International Conference on Evaluation and AsSoftware Quality, Reliability and Security Compan- sessment in Software Engineering, 2018, pp. 13–23. ion (QRS-C), IEEE, 2021, pp. 01–10. [30] J. Maxwell, Understanding and validity in qualita[17] P. Valenzuela-Toledo, A. Bergel, Evolution of tive research, Harvard educational review 62 (1992) 279–301.