-Maintaining a long-living software system is substantially related to the quality of the code the system is built from. In this experience report we describe how a set of practices and tools has been established and used on the early stages of a project. The approach is based on Clean Code and the use of well known static code analysis tools. The tools and practices have been used with an immediate effect of having cleaner code that is easier to understand in the long term. Additional attention is given to the cultural aspect that is involved in reaching a mindset that will allow to set and uphold code quality standards. Reaching a common understanding is a team effort that requires ”leaving one's comfort zone“. Finding common ground can significantly decide about failure or success in creating maintainable code.
Building long-living software involves many decisions that influence maintainability and extensibility of a system throughout its life-cycle. In early stages of building a softwaresystem the focus is often on a certain set of platform technologies and state of the art patterns. During its following lifetime the software-system will be challenged by inevitable change of requirements, increased workload triggering performance issues, modernization requests due to emerging technologies and the like.
Evolution and extension of a long-living software often
requires in-depth knowledge about the system which is only
present with senior engineers who have been working on
the system for a long time. Therefore, teams often have a
team historian whose knowledge is used whenever a source
code related question is raised [
With the focus mostly being put on technology and patterns in the early stage of a project, the code itself is given less attention. While allowing rapid movement and aggressive changes in the beginning, this will turn up later as hard to understand, maintain, and enhance code – effectively increasing the time and cost needed to accomplish the necessary evolution. This effect is commonly referred to as technical dept, a term coined
This research is funded by the Federal Ministry of Education and Research
(BMBF, Germany) in the Titan project (https://www.industrial-devops.org,
contract no. 01IS17084A–B).
by Cunningham [
There are plenty of good papers available on tools to analyze
code for its general quality and also to alert to potential
defects, for example [
Through the introduction of agile software development and
a cultural move towards DevOps, a versatile set of tool-chains
and practices has been established for rapid and constant
evolution of long-living software. We believe that combining
this with clean code [
We present an experience report on how achieving high
quality code has been implemented in the early stages of
the project “Titan – Industrial DevOps platform for iterative
process integration and automation” (Titan) [
The Titan project aims to explore and build the prototype of a scalable, resilient, and distributed software platform for small and medium enterprises (SME). Additionally, by transferring the principles of DevOps into the industrial domain, the platform is sided by the methods of Industrial DevOps. This will enable SMEs to use agile processes for a larger number of roles present even in small organizations. The project is executed as a cooperation between a software house and a university. The original proposal for the project already included the use of lean and agile methodology during the research and implementation phase. Also a focus was turned towards code quality and longevity of the platform.
The Titan platform itself will be made available as open source software. Open sourcing is aimed at reducing the vendor lock-in that comes with most other software platforms used by SMEs. On the other hand, open source software is tarnished by the perception of being less secure and of lower quality then commercial software. While the factor of quality and security cannot be validated for commercial software without gaining access to source code this insight is possible for open source.
From the very start of the Titan project an approach has been taken to enforce a route to produce readable and maintainable code. This approach is based on four factors which are described in the following.
The code used to build a software system is often written by software engineers which come from several backgrounds and have varying experience. Different parts of a system may use the same programming language or, as it is the case for the Titan project, multiple languages: 1) core libraries use C, 2) prototype backend processes Python or Java, 3) frontend applications JavaScript, TypeScript, and ECMAScript6. Open source software allows complete access to the code to understand or scrutinize the inner workings of the software. On the other hand it heavily relies on contributions from the outside to extend and modify it. Attracting new contributors thus is highly related to the code being understandable not only to its original author. Over time many bad experiences of developers with legacy code have led to the “Clean Code” movement which has put forth a set of practices and principles that ease the writing of good code.
Applying and enforcing these practices from the very start becomes a key point in avoiding technical dept which is hard to eliminate later on. Besides educating individual developers about the practices to use, our approach is sided by several other aspects. One of them is selecting or authoring “code style guides” appropriate to the programming language of a (sub)project. The guides are reviewed and agreed on by the core team of the project. Enforcing style is not limited to adhering to a fixed style of 1) code formatting, but also in 2) consistent use of language features e.g.: using or omitting the “this” statement in Java code, 3) naming conventions, and 4) use of meaningful names. The first two items can easily be checked and enforced in an automated way using static code analysis. This can happen inside an Integrated Development Environment (IDE) or in build pipelines of Continuous Integration environments. The latter two are only partially covered by analysis tools and are best addressed by making use of code review processes. Code that is committed to any of the repositories of the Titan project is cross-checked by a second developer before being accepted into the master branch. If code smells or non compliance are identified, the original developer and the reviewer enter into a discussion and resolve the problem by immediate refactoring. This requires a large amount of openness and trust between the parties involved in the discussion. The same openness and trust is also required throughout the whole project team. We experienced this to be a learning process with a rather steep learning curve and expect this to be likewise for most teams. The most prominent issue is overcoming the psychological aspect involved on constructive criticism and positive feedback.
Using Test Driven Development, where applicable, from the start forces developers into certain habits while coding. At the same time TDD lays the ground for later refactoring. Being forced to first write a test and then only write as much code as to make it pass the test will keep developers working in a very short loop. Classes and functions will be kept short and focused on the single task that they are intended to perform again delivering cleaner code. The availability of a test can be used straight away to refactor the very function that is currently being worked on, helping the developer in making changes without breaking the functionality.
In the long term the availability of comprehensive test
coverage will enable developers to perform larger refactoring [
Static analysis tools analyze the source code with regard to several aspects ranging from syntax checks to detecting coding mistakes and security vulnerabilities. Depending on the programming language, different sets of tools exist for static analysis. While many tools only target a single aspect of code quality, other cover multiple aspects. With the latter some overlap in their functionality can be found.
Even though tools that do static analysis of source code
have been around for a long time they are seldom used
with smaller or prototype projects [
We execute these static analysis tools during different phases of the software development process. Firstly, the software developer uses them while writing code. In addition to a manual, regular execution, we integrate the tools into the Integrated Development Environment if this is supported. Violations of coding guidelines or potential bugs are automatically highlighted in the source code. Secondly, the static analysis tools are executed during the local build process, for example, via build tools such as Gradle9. Both approaches together have been proven beneficial as they directly provide feedback on the quality of code and also serve as a guidance factor. However, making use of them can not strictly be enforced in this way. In order to ensure a certain degree of code quality in the common code base, defined quality gates have to be checked in a compulsory, automated, and uniform way as described in the following.
Automating tasks that are executed repeatedly and manually is one of the main objectives when employing DevOps principles. Directly accompanying the DevOps focus of the Titan project by making use of automation from the very beginning for its build environment is the logical choice in this case. CI also becomes the main step in enforcing code quality measures. In our approach the former mentioned static code analysis is performed as a build step in each of the subprojects build pipelines. Analysis is performed with the tool or tools matching the programming language of the subproject. Configurations for the static analysis tools are made part of the source code of the project. This is done in the form of configuration files for the static analyzers or directly in the steps of the declarative pipeline script.
Generally, using declarative pipeline job descriptions that are handled as checked-in code does enhance the transparency. Developers can change, add to, and adjust the build steps by making changes to the declaration. The configuration is not hidden in a separate system providing the build infrastructure. Through using GitLab10 as our choice for code hosting and CI/CD, building and running pipelines becomes a selfservice. Developers gain direct access by adding the declarative pipeline script to their project without further need for configuration in a separate interface of a CI/CD environment like Jenkins. Build pipelines that generate artifacts and deliver feedback to the developer are immediately triggered on the next push of code to the hosting platform.
Pipeline steps that provide quality gates should be set to fail, if any of the analysis tools reports flaws or metrics that exceed 6http://checkstyle.sourceforge.net 7https://pmd.github.io 8https://spotbugs.github.io 9https://gradle.org 10http://gitlab.org the thresholds defined, effectively forcing the developer to refactor the code right away. With the next improvement cycle the pipeline possibly can be completed again and artifacts that meet the set-out standards become available for deployment.
Using the approach described above, we observed
immediate improvement on code quality and use of secure coding
standards. Even though the Titan project is still in its early
stages this already became evident not only due to the metrics
used, but also during the manual reviews. First commits of
code did seldom meet initially set quality standards. This
observation became especially apparent when first running
GNU complexity on the C code subprojects. During the first
iterations of Titan subproject ujotypes-c11, for instance, 21
functions were reported being above a complexity level of 5,
the set threshold. The number of functions was lowered to 8
within a few iterations. The overall maximum complexity was
lowered from 37 to 13 for the most complex function. With
a complexity value of 37 being considered as: “Difficult to
maintain code”, whereas 13 is to be: “Maintained with little
trouble” [
In our Java-based subprojects12, static analysis tools were
not introduced until after an initial prototyping phase. In this
context, we also decided to apply established code formatting
guidelines. Using the tool Checkstyle forced us to comply with
the code style guidelines as all violations were reported as
errors. Besides formatting rules, PMD and Checkstyle identified,
for example, for the Control Center History subproject [
A main drawback in our opinion can be found in the
usability of the feedback provided by the static analysis tools and
the visibility of results in the CI/CD environment. The static
analysis tools reported flaws, insecure coding or exceeded
complexity but mostly gave little guidance as to how to counter
the reported issue. The lack of guidance leads to a situation
where individual developers repeatedly, and individually go on
a search for a solution to the same problem potentially with
a varying outcome. The same observation has been made by
Johnson et al. [
Based on our experiences during the early stages of the Titan project we presented an approach that improved source code quality from the beginning of software development on. Our goal of reaching clean code and essential maintainability in a long-living software platform is substantially supported. The project did not yet experience outside evolutionary pressure besides the internal iterative aspect of agile processes, because of this the chosen approach requires further validation throughout the project. None the less we expect using a set like the presented to help avoid degradation of maintainability. The choice of procedures and practices is more a cultural than a technical issue. There are plenty of tooling solutions available that have a long history and proven background. Making best use of them though requires a mindset of openness between developers to create acceptance. A common understanding has to be reached as to why code needs to adhere to committed quality standards. The DevOps culture and the agile software development practices provide a fruitful ground for employing the described approach.
Not only clean code aspects can degrade over time, but also
quantitative quality characteristics such as performance have
to be maintained for long-living software. As future work, we
therefore plan to integrate performance benchmarks into the
CI/CD infrastructure as suggested by Waller et al. [