A Quality Gate Role in a Software Delivery Pipeline Marko Gluhak and Luka Pavlič University of Maribor, Faculty of Electrical Engineering and Computer Science, Koroška cesta 46, Maribor, Slovenia Abstract This paper is in the area of DevOps, precisely, software delivery pipelines. It outlines the importance of an internal software quality for a long-term success of a software product due to its easier maintenance. By conducting a systematic literature review, we demonstrate, that quality gates in a form of an automatic source code analysis are an underutilized concept. In the paper, we present a case-study, demonstrating an automated quality gate. A prototype pipeline using GitHub Actions and GitLab CI/CD on an existing project, the paper also demonstrates a real-life challenges of implementing a quality gate. They primarily lie in philosophy changes. In addition, software metric thresholds and not straightforward to set. Keywords 1 Quality gate, DevOps, CI/CD pipelines, Static code analysis, GitHub Actions, GitLab CI/CD, SonarCloud. 1. Introduction A current state of the DevOps is constantly evolving [1], adding new testing layer. With each new layer we strive to contribute a more robust product. After several years of evolution in the field, the DevOps pipelines have contributed heavily to the software solutions development cycle enhancements. One of the most important goals of adding robustness to our product, is to facilitate increased confidence to the software release cycle [2]. Companies aspire to build pipelines that allow them to deploy their solutions quickly and with confidence in their stability. With such an aspiration in mind, many fail to pay attention to the product internal quality [3] – namely, the source code quality. Many test cycles merely cover the product external quality or rather, the aspects of the product behavior. The aspect addresses customers perspective, and covers nothing about how our developers perform [4], [5]. An importance of a quality gate on a source code level is understood to be vastly underutilized in the CI/CD pipelines for many software projects [3]. Companies, that can integrate this principle efficiently in their pipeline, might have more competitive advantage as a result [6], [7]. If internal product quality is neglected, we risk accumulation of technical debt [4]. It is a phenomenon which occurs when we sacrifice product quality, usually for the sake of delivering more functionalities. No matter the reason, the short-term outcome is low quality source code. Once we hit a point, where it is under acceptable level, the development can become slower and as a result, cost inefficient. To address this problem, companies adopt approaches such as code reviews, targeted static code analysis, linting etc. While, all of these are valid, they have an issue of relying on human consistency for effectiveness [8]. A key to success lies in establishing widely agreed rules, then are enforced in every development iteration. A quality gate in a CI/CD pipeline can be realized in the form of a static source code analysis. It offers a repeatable and a consistent internal quality analysis after every source code push to a version control system [3]. Our research was motivated by the role of quality gates in practice. This is why we formulated the following research questions:  RQ1: What is the role of the quality gate in the CI/CD pipeline? SQAMIA 2022: Workshop on Software Quality, Analysis, Monitoring, Improvement, and Applications, September 11--14, 2022, Novi Sad, Serbia EMAIL: marko.gluhak@student.um.si (M. Gluhak); luka.pavlic@um.si (L. Pavlic) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 6:1  RQ2: What are the difficulties, practitioners may face implementing a quality gate?  RQ3: What are the tools, that are most used for implementing a quality gate? We would like to outline a possible threat and a limitation of our research. In this paper we define the term “quality gate” as an internal software quality check, performed by static source code analysis. The rest of the paper is structured as follows. In the second chapter we present our research method and details on the systematic literature review. In chapter three, we present the systematic literature review results. The fourth chapter describe a quality gate application into practical case study. In the fifth chapter we focus on answering research questions. And finally, in the sixth chapter, we conclude our work with key insights, ending with acknowledgments and references. 2. Research Method To research the topic and research questions from the introduction, we’ve conducted a systematic literature review. Addressing all of the research questions with it, providing a descriptive answer for each. With the review we aim to understand what has currently been done in our research area. To better quantify the third research question, we will also be conducting a counting of tools mentioned in the papers. While not directly correlated to the principle of quality gates, this will give good indication of which tools are more commonly used. It is common that in practice we aim to use already known an established tools in newer concepts. Besides proving reliable in past experiences, they also provide some well needed familiarity when implementing new approaches to software development. The case study is aimed at verifying the findings in practice and enrichen insights of research questions one and two. It will be presented in further detail in the following sections of what we aim to achieve. 2.1. Systematic Literature Review The systematic literature review was carried out over four of the most popular scientific paper web libraries. They are IEEE, ScienceDirect, SpringerLink and ACM. We’ve crafted four search queries to address specific research aspects of our proposed principle. The exact search query term and its respective explanation is described in more detail in table 1. Table 1 Systematic literature review search queries and explanations ID Search query Explanation 1 "quality gate" AND ("devops" Finding connections between OR "software engineering") quality gates and automation AND "automation" 2 "quality gate" AND ("CI" OR Finding papers describing "CD" OR "CI/CD") quality gates in CI/CD pipelines 3 Finding studies about "quality gate" AND implementations and practical "automation" AND ("principles" examples of quality gate’s OR "implementation") usage 4 “quality gate" AND Finding the tools that help "automation" AND ("tool" OR facilitate the proposed "framework" OR "aid") principle. We’ve conducted the search for all four queries on the March the 9 th in 2022. The libraries yielded 721 papers for our further examination. A tabular representation of papers found by search queries and libraries is presented in table 2. 6:2 Table 2 Systematic literature review yielded papers by digital library and search query ID Library Number of papers 1 ACM 10 2 ACM 9 3 ACM 15 4 ACM 19 1 SpringerLink 110 2 SpringerLink 97 3 SpringerLink 117 4 SpringerLink 128 1 IEEE 1 2 IEEE 0 3 IEEE 1 4 IEEE 1 1 ScienceDirect 24 2 ScienceDirect 54 3 ScienceDirect 68 4 ScienceDirect 67 We’ve then analyzed the papers in a systematic manner, following the inclusion and exclusion criteria. The process of analyzing took 6 phases to complete before the primary research papers were selected. They consisted of the following steps.  First phase: digital library search. With the prepared queries presented in table 1 we queried the digital libraries. We have displayed the results in table 2. We have acquired a total of 721 papers.  Second phase: removal of duplicates. Here, we have conducted a simple check of whether a paper has duplicates in other digital libraries and removed them if so. There are 290 papers left after this step.  Third phase: review by title and keywords. On this step, we have carefully examined the title and keywords of a papers, considering its worthwhileness to our goal of answering the proposed research questions. If the paper seemed useful or we were uncertain of its usefulness it advanced to the next phase. Thirty-six papers remained.  Fourth phase: review by abstract. This phase consisted of us examining and considering the content of the papers abstract. Upon which we concluded whether it was useful or not. If useful or uncertain about its usefulness we added the paper to the next phase. Twenty-three papers remained.  Fifth phase: full paper review. At this stage we fully examined the content of a paper and carefully examined whether it helps us address the research prospects. If so, we added it to the selection of primary paper candidates. If we were uncertain at this stage, we removed the paper from further examination. Nineteen papers remained.  Sixth phase: primary research selection. Here we already had a base of nineteen papers which we added from phase 5. Henceforth we looked at their references, browsed the web for other sources of knowledge. We found six contributions of interest to our research. After the sixth phase we arrived at the total number of twenty-five primary research sources. We’ll analyse them in further detail in the following chapter. 6:3 3. Results In this section we shall present the results of the systematic literature review. Upon its completion we discover that the fields of DevOps, CI/CD pipelines and traditional quality assurance matured in their own regard. The area of static code analysis also gained prevalence over the years. But putting the fields together in the form of an automated quality gate is a more recent phenomenon. Thus, it warrants further study and inspection of how it is and can be further utilized to the benefit of software solution’s quality. The papers chosen for a primary base of knowledge is displayed in table 2. Table 2 Resulting publications of the systematic literature review. ID Title Authors A longitudinal study of static analysis warning evolution and the 1 A. Trautsch, S. Herbold, and J. Grabowski effects of PMD on software quality in Apache open source projects Agile Development Offers the Chance to Establish Automated Quality 2 M. Kösling and A. Poth Procedures An empirical characterization of bad practices in continuous F. Zampetti, C. Vassallo, S. Panichella, G. 3 integration Canfora, H. Gall, and M. di Penta An empirical study on self-admitted technical debt in modern code 4 Y. Kashiwa et al review Challenges and solutions when adopting DevSecOps: A systematic R. N. Rajapakse, M. Zahedi, M. A. Babar, and 5 review H. Shen Challenges in Adopting Continuous Delivery and DevOps in a Globally R. K. Gupta, M. Venkatachalapathy, and F. K. 6 Distributed Product Team: A Case Study of a Healthcare Organization Jeberla Comparison of release engineering practices in a large mature E. Laukkanen, M. Paasivaara, J. Itkonen, and 7 company and a startup C. Lassenius Comparison of Static Analysis Tools for Quality Measurement of RPG 8 Z. Tòth, L. Vidàcs, and R. Ferenc Programs T. Amanatidis, N. Mittas, A. Moschou, A. Evaluating the agreement among technical debt measurement tools: 9 Chatzigeorgiou, A. Ampatzoglou, and L. building an empirical benchmark of technical debt liabilities Angelis Exploration of DevOps testing process capabilities: An ISM and fuzzy S. Rafi, M. A. Akbar, W. Yu, A. Alsanad, A. 10 TOPSIS analysis Gumaei, and M. U. Sarwar Improving students’ programming quality with the continuous 11 Y. Lu, X. Mao, T. Wang, G. Yin, and Z. Li inspection process: a social coding perspective Introduction of static quality analysis in small- and medium-sized M. Gleirscher, D. Golubitskiy, M. Irlbeck, 12 software enterprises: experiences from technology transfer and S. Wagner M. M. Ahmad Ibrahim, S. M. Syed- 13 Managing Quality Assurance Challenges of DevOps through Analytics Mohamad, and M. H. Husin On the Benefit of Automated Static Analysis for Small and Medium- M. Gleirscher, D. Golubitskiy, M. Irlbeck, 14 Sized Software Enterprises and S. Wagner Strategies to manage quality requirements in agile software 15 P. Karhapää et al development: a multiple case study A. Janus, R. Dumke, A. Schmietendorf, and J. 16 The 3C Approach for Agile Quality Assurance Jäger 17 The Professional-Grade DevOps Environment J. Palermo R. Pietrantuono, A. Bertolino, G. de Angelis, 18 Towards Continuous Software Reliability Testing in DevOps B. Miranda, and S. Russo G. Schermann, J. Cito, P. Leitner, and H. C. 19 Towards quality gates in continuous delivery and deployment Gall Are Static Analysis Violations Really Fixed? A Closer Look at Realistic D. Marcilio, R. Bonifacio, E. Monteiro, E. 20 Usage of SonarQube Canedo, W. Luz, and G. Pinto Beyond Continuous Delivery: An Empirical Investigation of Continuous M. Shahin, M. A. Babar, M. Zahedi, and L. 21 Deployment Challenges Zhu Carrot and Stick approaches revisited when managing Technical Debt Y. Crespo, A. Gonzalez-Escribano, and M. 22 in educational context Piattini Code quality & Quality gates Evaluating efficiency and usability of 23 three activities for improving code quality EDAN80-Coaching of F. Nyberg and J. Skogeby programming teams In-depth study End to End Automation On Cloud with Build Pipeline The case for 24 M. Soni DevOps in Insurance Industry 6:4 25 The Use of the Software Metrics in Practice L. Pavlič, M. Okorn, And M. Heričko To keep in line with our established research questions, we kept closer attention to the tools mentioned in research papers regarding static code analysis. To better justify our main addition to the pipeline in the form of static code analysis we tracked the tools mentioning across the found papers. The results can be observed in table 3. Table 3 Results of the tools mentioned across papers tracking Tool Number of mentions Paper’s IDs 1, 2, 3, 8, 9, 11, 14, 17, 20, 22, SonarQube 12 23, 24 SonarLint 1 23 PMD 7 1, 9, 12, 14, 16, 20, 23 JArchitect 1 22 Codacy 1 23 FindBugs 7 1, 9, 12, 14, 20, 23, 24 Squore 1 9 CAST 1 9 SonarJ 1 12 Gendarme 1 14 Checkstyle 2 16, 23 NDpenend 1 22 From our tracking activity SonarQube is the most mentioned tool across all our acquired papers in the systematic literature review. It is also worth mentioning that SonarJ and SonarLint found in our papers belong to the same family as SonarQube, but in other forms of integration. Therefore, increasing the number of mentions further than just the twelve. PMD and FindBugs, both share seven respective mentions for a tied second place. 4. Case study: A CI/CD pipeline To support our theoretical findings from the systematic literature review we conducted a practical example of a CI/CD pipeline and complementing it with a quality gate in the form of automated static code analysis. The goal was to answer the practical aspect of research questions better. To better understand wherein the true difficulty of adopting the principle of automated static code analysis as quality gates in the software solutions delivery pipeline. We set out to prototype a simple yet representative pipeline, which would include the building and testing the pushed code to a repository. Then added a quality gate in the form of static code analysis to it. We present the pipeline design in figure 1. Once we push new code to the GitHub repository building and analysing is done via GitHub Actions. Upon completion of these test our added step of static source code analysis takes place in SonarCloud. If all these steps succeed, we deploy the software solution to production in line with our setup pipeline. 6:5 Figure 1: Flow of the software solutions CI/CD pipeline. We’ve chosen the following technologies for analyzing our pipeline. All decisions are based on preliminary results of the systematic literature review. We’ve chosen two providers of CI/CD pipelines, to attempt at better generalizing our findings to the real-world use cases at implementing such ideas to already existing and newly established pipelines. GitHub is arguably one of the most popular platforms in terms of version control and offers a robust platform for conducting CI/CD tasks in the form of GitHub Actions. Although it did not start primarily as a CI/CD tool, it has proved to be effective at this. It requires a YAML specification of a task and then runs it upon the specified ruleset. It should also provide a contrasting candidate for our other choice of CI/CD provider [9]–[11]. GitLab is also a very popular platform for managing version control. The first major difference to GitHub that is owned by Microsoft is that it is open source. Another difference lends itself in terms of its Gitlab CI/CD which is purpose built for the task of creating and operating continuous integration and continuous deployment pipelines. Although requiring the same YAML specification to build pipelines it is of a different syntax [12]–[15]. SonarQube is one of the most comprehensive tools for static code analysis. It not only analyzes the quality of said code its technical debt, but also scans for other major security vulnerabilities. Which bridges the gap between DevOps and DevSecOps. Making the results of our research a bit richer and comparative to the real world. It also provides a simple solution for utilization in CI/CD pipelines to allow for easier analysis of procured results [16]. Due to pricing and features offered, we use a cloud variation of SonarQube – SonarCloud. 4.1. Project under test The project used for conducting our case study is a student project, developed by a group of four students over the course of three years, beginning in June of 2019. The project is of a smaller size, counting about six thousand lines of code using the following technologies:  ReactJS,  ExpressJS,  MongoDB,  ChaiJS,  GitHub Actions,  Python. The project is mostly written in the JavaScript programming language with Python supporting the machine learning aspects of it. The project is deployed on the following link, https://varno-domov.si. Source code can be viewed on GitHub at the following link, https://github.com/MerceneX/BoomMap. The architecture is simple, consisting of three separate parts, communicating with each other 6:6 (frontend single page application, backend server and database). All of it is hosted on FERI container services. The domain and DNS are outside providers Domovanje and Free DNS – observable on figure 2. Figure 2: Architecture of the “Varno Domov” project, used for the purposes of the case study. Current state of the CI/CD pipeline is rather basic in the form of GitHub Actions taking care of three key steps in testing the reliability of the software solution. It is using a matrix strategy to test different Node versions and is being ran on every “push” and “pull request” event. The activities being covered in the steps are as follows: 1. “npm ci” is being ran first, facilitating a clean installation of packets and libraries required for the project to run. 2. “npm run build –if-present” builds our software solution to check if it may be run with the installed libraries and packets 3. “npm run test” takes the code into action and runs the written tests for the code to take place, analyzing if the behavior is as designed. With these steps we established a simple pipeline with quality checks dependent on each other to succeed. If one is to fail, the pipeline fails, and code is not integrated to the larger codebase. We are confident that the project with its variable code quality, used platforms and planned additions will provide a good base for our case study and make insights gathered generalizable for further study. 6:7 4.2. The Quality gate effects Upon the addition of the quality gate in the form of SonarCloud’s static source code analysis we decided to set a quality gate on new code and not the entire codebase. This is one of the good practices found in the systematic literature review, for it ensures a smoother transition in a new principle’s implementation. We can observe the results of the first analysis in figure 3. The added new code was sufficiently written to adhere to the new quality standards. Figure 3: First analysis from SonarCloud and quality gate information One of the discovered pitfalls can be upon implementation of the quality gate is to set it too strict and on the entire codebase. This can lead to loss of morale among developers when writing code and attempts in cheating the quality gate. The standards should incrementally increase the quality of new code. Thus, increasing the quality of overall code via inevitable alterations in older pieces of code. If quality is set too strict it is almost the same as enforcing an entire rewrite of the codebase. This act, although necessary in some cases, also led many projects to their ruin and the decision of a rewrite should not fall to a single quality gate result. During our case study, we discovered that if the quality gate is set so that it is feasible to be obeyed, the resulting quality does indeed increase. Although after initial implementation of the quality gate it took, some getting used to, it was manageable and useful. To avoid failings of quality gate an aid like SonarLint might be used to contribute to developer satisfaction. 5. Discussion To draw conclusions from our research, we begin by answering research questions set in the introduction, to guide our discussion further. 5.1. RQ1: What is the role of the quality gate in the CI/CD pipeline? Current research on the topic is still fresh. It suggests that the role of the quality gate specified in more detail on a case-by-case basis. That is, when even used. Although companies already have a robust CI/CD pipeline, they do not conduct static code analysis in it, leaving internal quality checks 6:8 up to developer or have manual quality gates in place e.g., code reviews. Contrary to intuition, the most common reason for companies not adopting automated quality checks is the worry of quality. Distrust towards tools such as SonarQube, false positives and other wariness towards attitude shifts in assuring quality are all valid concerns. Although not as commonly practiced of a principle as we had hoped for, there are still two distinguishable types of practitioners that present themselves upon inspection of the papers: a. Larger companies, which have robust and comprehensive pipelines already in place have a fondness of adding quality checks as possible at every step of the development process. Although assuring quality and having added value it is worth noting that if these checks become too demanding and constraining, they may hinder their own purpose or even the entire development process. Developers may also attempt to look for shortcuts on how to overcome these quality gates. b. Smaller companies prove to be more successful in practicing the static code analysis quality gate. The concept lends itself better to cloud oriented pipeline services. Worth mentioning is also the fact that it is easier to overcome the attitude barrier towards a new paradigm in projects inception phases rather than down the line when code base becomes less known. 5.2. RQ2: What are the difficulties practitioners may face implementing this principle? The main difficulty is a psychological one as we have uncovered by our systematic literature review. This notion confirmed by the fact that in our prototypical implementation of a quality gate in a CI/CD pipeline is simple. Regardless of the fact if we are adding it to an existing pipeline or implementing it in a new one. The difference between platforms GitLab CI/CD and GitHub Actions is miniscule enough, to not even warrant a proper comparison. We can draw the decision on which to use up to personal preference of GUI and YAML syntax rules. At least for the purpose of implementing quality gates. The true difficulties lie in the details and general acceptance of the new paradigm in quality assurance. Most companies list quality as the top concern for not implementing automated quality checks in the form of quality gates. They use code reviews and targeted manual static code analysis. Although a valid concern, because of shortcomings in tools like SonarQube in the form of too many false positives, unrealistic technical debt estimation… It is our finding, that even if these tools do not accurately assess the state of quality, they do serve to increase it. Setting quality gates on a project-by-project basis appears as the correct way to be. Which also makes sense from a business perspective since a project worth more should be of higher quality. Thus, the challenge is figuring out the correct settings in said quality gates, to reflect the quality expected for that project. And code metrics thresholds are a whole other topic, out of the scope of this paper, but will play a key role in the effectiveness of set quality gates. 5.3. RQ3: Which tools are most used for implementing this principle? As gathered from our systematic literature review, the most mentioned tool is SonarQube. It supports all the functionalities for a successful static code analysis, it allows project by project configuration of quality gates. That is one of the better practices identified as well, to tailor every project as a case-by-case. Another reason for SonarQube’s popularity is without a doubt the ability to integrate the tool at every step of the development process. The second most mentions belong to PMD which is also a generic tool for static code analysis and FindBugs, which is more language focused. On the topic of platforms that offer CI/CD services there is little to no discussion or mention. Names like Azure and Jenkins come up, with no further elaboration. We assume that larger companies prefer tools that offer more adjustability to their use case an opt for on premise teams for support. Smaller ones prefer to keep it simple, and cloud based. 6:9 6. Conclusion In the end we find ourselves concluding our research with a more thorough understanding of what a quality gate in software delivery pipeline is meant to be – a developer’s companion in the search for higher quality. It is there to support decision making and encourage questioning decisions that seem poor in terms of code quality. Although false positives and disparaging estimations of technical debt may undermine this goal, it is however still a boon to use these features. In our systematic literature review we found that the most efficient way to use this principle is with general guidelines and not robust and overly strict rule that fail builds constantly. The principle also aims to improve the mentality shift towards quality assurance and developing cleaner code. Which is still the biggest concern with implementing this principle, that it does not become a hinderance and falls to the side. 7. Acknowledgements The authors acknowledge financial support from the Slovenian Research Agency (Research Core Funding No. P2- 0057) 8. References [1] A. K K, “DevOps Basics and Variations,” Azure DevOps for Web Developers, pp. 1–11, 2020, doi: 10.1007/978-1-4842-6412-6_1. [2] R. Pietrantuono, A. Bertolino, G. de Angelis, B. Miranda, and S. Russo, “Towards continuous software reliability testing in DevOPs,” Proceedings - 2019 IEEE/ACM 14th International Workshop on Automation of Software Test, AST 2019, pp. 21–27, May 2019, doi: 10.1109/AST.2019.00009. [3] S. Rafi, M. A. Akbar, W. Yu, A. Alsanad, A. Gumaei, and M. U. Sarwar, “Exploration of DevOps testing process capabilities: An ISM and fuzzy TOPSIS analysis,” Applied Soft Computing, vol. 116, p. 108377, Feb. 2022, doi: 10.1016/J.ASOC.2021.108377. [4] Y. Kashiwa et al., “An empirical study on self-admitted technical debt in modern code review,” Information and Software Technology, vol. 146, p. 106855, Jun. 2022, doi: 10.1016/J.INFSOF.2022.106855. [5] Y. Crespo, A. Gonzalez-Escribano, and M. Piattini, “Carrot and Stick approaches revisited when managing Technical Debt in an educational context,” Proceedings - 2021 IEEE/ACM International Conference on Technical Debt, TechDebt 2021, pp. 99–108, May 2021, doi: 10.1109/TECHDEBT52882.2021.00020. [6] M. Shahin, M. A. Babar, M. Zahedi, and L. Zhu, “Beyond Continuous Delivery: An Empirical Investigation of Continuous Deployment Challenges,” International Symposium on Empirical Software Engineering and Measurement, vol. 2017-November, pp. 111–120, Dec. 2017, doi: 10.1109/ESEM.2017.18. [7] E. Laukkanen, M. Paasivaara, J. Itkonen, and C. Lassenius, “Comparison of release engineering practices in a large mature company and a startup,” Empirical Software Engineering, vol. 23, no. 6, pp. 3535–3577, Dec. 2018, doi: 10.1007/S10664-018-9616- 7/TABLES/14. [8] A. Janus, R. Dumke, A. Schmietendorf, and J. Jäger, “The 3C approach for Agile Quality Assurance,” 2012 3rd International Workshop on Emerging Trends in Software Metrics, WETSoM 2012 - Proceedings, pp. 9–13, 2012, doi: 10.1109/WETSOM.2012.6226998. [9] “Features • GitHub Actions · GitHub.” https://github.com/features/actions (accessed Feb. 05, 2022). [10] “GitHub Code & Quality Analysis | GitHub Integration | SonarQube.” https://www.sonarqube.org/github-integration/ (accessed Feb. 05, 2022). [11] “Create a quality gate with GitHub | Clayton Help Center.” https://help.clayton.io/en/articles/2152713-create-a-quality-gate-with-github (accessed Feb. 05, 2022). 6:10 [12] “GitLab CI/CD | GitLab.” https://docs.gitlab.com/ee/ci/ (accessed May 18, 2022). [13] “GitLab Code & Quality Analysis | GitLab Integration | SonarQube.” https://www.sonarqube.org/gitlab-integration/ (accessed Feb. 05, 2022). [14] “CI/CD pipelines | GitLab.” https://docs.gitlab.com/ee/ci/pipelines/ (accessed Feb. 05, 2022). [15] “Create a quality gate with GitLab | Clayton Help Center.” https://help.clayton.io/en/articles/2871439-create-a-quality-gate-with-gitlab (accessed Feb. 05, 2022). [16] “Code Quality and Code Security | SonarQube.” https://www.sonarqube.org/ (accessed Feb. 05, 2022). [17] A. Trautsch, S. Herbold, and J. Grabowski, “A longitudinal study of static analysis warning evolution and the effects of PMD on software quality in Apache open source projects,” Empirical Software Engineering, vol. 25, no. 6, pp. 5137–5192, Nov. 2020, doi: 10.1007/S10664-020-09880-1/TABLES/21. [18] M. Kösling and A. Poth, “Agile Development Offers the Chance to Establish Automated Quality Procedures,” Communications in Computer and Information Science, vol. 748, pp. 495–503, 2017, doi: 10.1007/978-3-319-64218-5_40. [19] F. Zampetti, C. Vassallo, S. Panichella, G. Canfora, H. Gall, and M. di Penta, “An empirical characterization of bad practices in continuous integration,” Empirical Software Engineering, vol. 25, no. 2, pp. 1095–1135, Mar. 2020, doi: 10.1007/S10664-019-09785-8/TABLES/13. [20] R. N. Rajapakse, M. Zahedi, M. A. Babar, and H. Shen, “Challenges and solutions when adopting DevSecOps: A systematic review,” Information and Software Technology, vol. 141, p. 106700, Jan. 2022, doi: 10.1016/J.INFSOF.2021.106700. [21] R. K. Gupta, M. Venkatachalapathy, and F. K. Jeberla, “Challenges in Adopting Continuous Delivery and DevOps in a Globally Distributed Product Team: A Case Study of a Healthcare Organization,” Proceedings - 2019 ACM/IEEE 14th International Conference on Global Software Engineering, ICGSE 2019, pp. 30–34, May 2019, doi: 10.1109/ICGSE.2019.00020. [22] Z. Tòth, L. Vidàcs, and R. Ferenc, “Comparison of Static Analysis Tools for Quality Measurement of RPG Programs,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9159, pp. 177–192, 2015, doi: 10.1007/978-3-319-21413-9_13. [23] T. Amanatidis, N. Mittas, A. Moschou, A. Chatzigeorgiou, A. Ampatzoglou, and L. Angelis, “Evaluating the agreement among technical debt measurement tools: building an empirical benchmark of technical debt liabilities,” Empirical Software Engineering, vol. 25, no. 5, pp. 4161–4204, Sep. 2020, doi: 10.1007/S10664-020-09869-W/TABLES/8. [24] Y. Lu, X. Mao, T. Wang, G. Yin, and Z. Li, “Improving students’ programming quality with the continuous inspection process: a social coding perspective,” Frontiers of Computer Science 2019 14:5, vol. 14, no. 5, pp. 1–18, Jan. 2020, doi: 10.1007/S11704-019-9023-2. [25] M. Gleirscher, D. Golubitskiy, M. Irlbeck, and S. Wagner, “Introduction of static quality analysis in small- and medium-sized software enterprises: experiences from technology transfer,” Software Quality Journal, vol. 22, no. 3, pp. 499–542, Sep. 2014, doi: 10.1007/S11219-013-9217-Z/TABLES/13. [26] M. M. Ahmad Ibrahim, S. M. Syed-Mohamad, and M. H. Husin, “Managing quality assurance challenges of Devops through analytics,” ACM International Conference Proceeding Series, vol. Part F147956, pp. 194–198, 2019, doi: 10.1145/3316615.3316670. [27] M. Gleirscher, D. Golubitskiy, M. Irlbeck, and S. Wagner, “On the Benefit of Automated Static Analysis for Small and Medium-Sized Software Enterprises,” Lecture Notes in Business Information Processing, vol. 94 LNBIP, pp. 14–38, 2012, doi: 10.1007/978-3-642-27213-4_3. [28] P. Karhapää et al., “Strategies to manage quality requirements in agile software development: a multiple case study,” Empirical Software Engineering, vol. 26, no. 2, pp. 1–59, Mar. 2021, doi: 10.1007/S10664-020-09903-X/FIGURES/6. [29] J. Palermo, “The Professional-Grade DevOps Environment,” .NET DevOps for Azure, pp. 71– 94, 2019, doi: 10.1007/978-1-4842-5343-4_3. [30] G. Schermann, J. Cito, P. Leitner, and H. C. Gall, “Towards quality gates in continuous delivery and deployment,” IEEE International Conference on Program Comprehension, vol. 2016-July, Jul. 2016, doi: 10.1109/ICPC.2016.7503737. 6:11 [31] D. Marcilio, R. Bonifacio, E. Monteiro, E. Canedo, W. Luz, and G. Pinto, “Are static analysis violations really fixed? a closer look at realistic usage of sonarqube,” IEEE International Conference on Program Comprehension, vol. 2019-May, pp. 209–219, May 2019, doi: 10.1109/ICPC.2019.00040. [32] M. Shahin, M. A. Babar, M. Zahedi, and L. Zhu, “Beyond Continuous Delivery: An Empirical Investigation of Continuous Deployment Challenges,” International Symposium on Empirical Software Engineering and Measurement, vol. 2017-November, pp. 111–120, Dec. 2017, doi: 10.1109/ESEM.2017.18. [33] F. Nyberg and J. Skogeby, “Code quality & Quality gates Evaluating efficiency and usability of three activities for improving code quality EDAN80-Coaching of programming teams In- depth study,” 2019. [34] M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” Proceedings - 2015 IEEE International Conference on Cloud Computing in Emerging Markets, CCEM 2015, pp. 85–89, Mar. 2016, doi: 10.1109/CCEM.2015.29. [35] L. PAVLIČ, M. OKORN, and M. HERIČKO, “The Use of the Software Metrics in Practice,” SQAMIA, vol. 2217, 2018. Source (Trauts ch et al., 20 20)(T rauts ch et al., 20 20) (Kösling & Poth, 2017)(Kösling & Poth, 2017) (Zampet ti et al., 2020 )(Zampe tti e t al., 2020) (Kas hiwa et al., 2022)( Kas hiwa et al., 2022) (Rajapaks e et al., 2022) (Rajapaks e et al., 2022 ) (G upta et al., 2019 )(G upta e t al., 2019) (Laukkanen et al., 2018) (Laukkanen et al., 20 18) (Tòth et al., 2015)(T òth e t al., 2015) (Amana tidis et al., 2020) (Ama natidis et al., 2020 ) (Rafi et al., 202 2)(Ra fi et al., 2022 ) (Lu et al., 2020) (Lu et al., 2020) (G leirs cher et al., 2014) (G leirs cher et al., 20 14) (Ahmad I brahim e t al., 2019 )(Ah mad Ib rahim et al., 2 019) (G leirs cher et al., 2012) (G leirs cher et al., 20 12) (Karhapää e t al., 2021) (Karha pää et al., 2021 ) (Janus et al., 2012 )(Jan us et al., 2012) (Palermo, 2019)( Palerm o, 2019) (Pietran tuon o et al., 2019 )(Pie tran tuono e t al., 2019) (Schermann e t al., 2016) (Scherma nn et al., 201 6) (Marcilio e t al., 2019)( Marcilio e t al., 2019) (Shahin et al., 2017 b)(Shahi n et al., 2017 b) (Cres po et al., 2 021)( Cres po e t al., 2021) (Nyberg & Skogeby, 2019)(Nyberg & Skogeby, 2019) (Soni, 2016)(S oni, 2016) (PAVLIČ e t al., 2018) (PAVLIČ e t al., 2018 ) 6:12