-Recent studies show that security vulnerabilities are caused by neglecting best-practices for the configuration of software and the underlying infrastructure. Due to the rising complexity of software systems and the accelerated speed of software releases using mechanisms like continuous delivery the problem gets even more challenging. Existing processes and methods are not adequate to cope with these challenges. This paper proposes an approach for continuous compliance testing. Using well-known methods from software testing, this approach enables an organization to define, organize, and execute compliance tests in a structured and reusable way. We focus in our approach onto integrating a software-centric point of view for modeling compliance requirements. By embedding our approach into a deployment pipeline automated continuous compliance testing can be realized. Index Terms-software testing, system testing, continuous software engineering, compliance, security, devops 1www.cisecurity.org 2www.nist.gov 3www.bsi.de 4International Software Testing Qualifications Board
Software must run on a defined operating infrastructure
to fulfill its purpose. In recent years, the assessment of
infrastructure gained more and more attention from a security
perspective. OWASP identified the Misconfiguration of Server
as one of the top10 security risks [
Therefore, Organizations like the Center of Internet Security (CIS)1, NIST2 and the German BSI3 developed and published tools, guidelines, checklists and benchmarks, which offer the opportunity to assess a system.
To assess the operating infrastructure, so called compliance
tests need to be performed. Lewis and Bassetti [
This paper is motivated by a lack of a structured and
reliable way to organize and assess compliance aspects, which
is intensified due to increasing software and infrastructure
complexity. Beside this, the challenges of growing pace in
software development through agile and automated delivery
of new functionality and software, are inevitable. Concepts
and methods like continuous delivery [
Otherwise, the compliance status of the operating infrastructure will erode, which leads to system misconfiguration and security vulnerabilities. In general a system misconfiguration has serious impact on the system and software security as stated in the OWASP Top 10 report. Thus the automation of compliance testing is necessary.
Besides testing the operating infrastructure for global standards and regulations, the operating infrastructure has to be compliant to the software and its configuration. Therefore, from a software perspective new compliance requirements are introduced. Any software vendor should be interested, that their software products and their customers infrastructure are compliant. For example, if a software component requires a special cryptographic library to provide sufficient security, it is necessary to check if this requirement is satisfied before bringing the software into operation. As a consequence, the infrastructure provided by the customer needs to be tested for compliance. Therefore, the vendor should be able to define custom compliance requirements for single or multiple related software components. Consequently, these specific software compliance requirements need to be tested as well, before deployment, after a deployment and during operation. Basically, compliance needs to be ensured continuously..
Although, it is easy to define certain compliance requirements on small scale, it gets more difficult when enlarging the scope towards complex enterprise systems. Hence a modular approach of defining, assigning, and checking those compliance requirements as already established for various standards from a software-centric perspective is needed.
This paper introduces an novel approach to fill this gap. The main contribution is an unified model for specifying, managing and executing compliance tests with a softwarecentric viewpoint. It supports the stakeholders to ensure, that
Fig. 1: Data-Driven Testing [
The paper is structured as follows. The next section will introduce the needed background of continuous delivery and data-driven testing. In section 3 will define the concept of a software-centric point of view for compliance testing. Based on theses results we introduce our model for continuous compliance testing. The subsequent section will deal with some important aspects as reusability and validation and the integration into DevOps. COMET, a tool implementing our approach, will be briefly described in the next section, followed by a short discussion about related work and existing tools for compliance testing. A final section will provide the conclusion and some proposals for future research in this area.
Data-Driven Testing (DDT) [
Humble and Farley [
Continuous delivery uses an automated development infrastructure, called deployment pipeline, which automates nearly every step of the delivery process. Each commit of a developer enters the deployment pipeline and an automated process is started, which produces a new software increment as a result artifact.
The deployment pipeline incorporates all activities known
from continuous integration [
If the software is directly used in an production environment
this extension is called continuous deployment. Continuous
delivery favors the principle of shift left [
Compliance testing and continuous delivery are compatible to extend each others functionality and usefulness. The following section will introduce an approach to specify these compliance requirements for environments. Finally, we present a tool to run compliance tests using data-driven testing in a continuously manner.
III. COMPLIANCE TESTING FROM A SOFTWARE-CENTRIC
POINT OF VIEW
The existing approaches to compliance testing like the CIS Benchmarks or the guidelines published by BSI target low level and minimal requirements for secure infrastructures. For companies and the affected stakeholders like the Chief Security Office (CSO) or the Chief Information Officer, it is hard to apply these guidelines to their current set of software systems. In general, most organizations apply the basic levels. We argue, that this level of compliance testing needs to be improved by extending the viewpoint from the infrastructure to the software systems and complete landscapes running on these infrastructures. We propose to shift to a software-centric perspective. Each software introduces additional individual requirements for the configuration of infrastructure and even of related software. These requirements need to be considered when conducting a compliance test. The following example explains the purpose and the need for a software-centric point of view. Consider a software system with the following requirements regarding the web server software Apache httpd5
running on a Linux server node: • The software Apache httpd runs as the user www-data. • Apache httpd uses the local TCP port 80 to provide the web pages to external clients via HTTP. • The served web pages are stored in a directory which can be specified with the property DOCUMENT ROOT. • This directory needs to be readable and writable. • It writes its logs to the directory /var/log/apache2. From these requirements we can deduce new configuration related requirements which need to be fulfilled to be able to run the Apache httpd software. Failing one of these requirements will result in a non-operational system. So, these requirements add new rules, to ensure that the system is configured correctly and compliant to the software.
Software systems nowadays do not consist of one software component but of multiple. Complex software applications form a distributed landscape of software components running on different nodes in the network. These landscapes can be specified and compiled dynamically and will change during the application lifecycle.
Consequently, from a software point of view all the configuration requirements of the different software components need to be considered when performing a compliance test for this software system.
Based on these insights, the following sections introduce a well-defined domain model for compliance testing which incorporates a software-centric point of view.
The proposed Continuous Compliance Model can be seen as
a domain model for continuous compliance testing. Following
the principles of Domain-Driven Design [
Figure 2 visualizes the continuous compliance. The contexts Operation and Software represent the software-centric point of view. Compliance inhabits concepts regarding compliance rules and their organization. The Technical context deals with the implementation of assessments as parameterized tests. The Execution context contains concepts necessary to perform, evaluate, and save tests and their results. The following sections will elaborate on the identified concepts and their relationships.
A Compliance Rule defines the shape of a specific aspect or property of the SUT which needs to be checked. If the rule is fulfilled the system is compliant regarding this specific rule. A compliance rule can comprise multiple varying Compliance Tests to assess the desired property.
Examples of compliance rules for a secure web service could be: 1) The SUT provides a service on the local port TCP 443. 2) The SUT uses the protocol HTTP to communicate on port 443. 3) The SUT offers encrypted communication secured by SSL/TLS on Port 443 with a minimal version of TLS 1.2. 4) The SUT uses a valid SSL certificate to encrypt its communication on port 443.
A Compliance Rule Set is a collection of specific compliance rules which belong together and need to be checked together. The examples from above can be aggregated to a rule set for checking a web server.
As compliance rules can be very specific, this leads to
a reduced reuse potential of this rules. Therefore, we
introduce variables which enable and support reuse of rules. A
compliance rule becomes parameterized in the same way as
parametrized tests [
VAR PORT. 2) The SUT uses the protocol HTTP to communicate on port VAR PORT. 3) The SUT offers encrypted communication secured by SSL/TLS on port VAR PORT with a minimal version of TLS 1.2. 4) The SUT uses a valid SSL certificate to encrypt its communication on port VAR PORT.
The variable VAR PORT is specific to the current SUT and can vary for different systems and/or different software components. The version of TLS could be also transformed into a variable, but a compliance rule should be defined as precise as possible (e.g., we do not want to allow to change the TLS version to on older less secure one). Organization wide policies can be applied at this level, the need for more context specific data (as a port number) can be delegated to higher level software components.
The relation between compliance rules and software components will considered closer in a subsequent section.
To check for compliance we need an automated way to evaluate compliance rules and apply them to actual systems and environments. Our approach to compliance testing differentiates between compliance rules and the way they are evaluated. This is represented by the concept of a Compliance Test. A compliance test is an implementation of an assessment of system properties. A compliance test can be executed and evaluated automatically. It offers a defined set of variables, which needs to be assigned before execution. The final test code will be derived and generated during execution of the test.
The technology used to evaluate the required properties for a compliance rule, can be chosen freely. Hence, the technology is not important and our approach is therefore technology agnostic.
Listing 1: Parameterized Compliance Test with InSpec c o n t r o l ” T e s t F o r O p e n P o r t ” do i m p a c t 1 . 0 Software Compliance Technical
Software Component Compliance
Rule Set
Component
Landscape
Compliance Rule Parameter
Compliance Test System
Environment
Execution
Test Execution
Report Test Execution
Generated Test Case d e s c r i b e p o r t ( VAR PORT ) do
i t { s h o u l d b e l i s t e n i n g } e n d e n d
Listing 1 defines a compliance test using InSpec6 to check if a certain port is used.
A compliance test can be used by a compliance rule to asses for its required system properties. A compliance rule can use multiple compliance tests. As the implementation of compliance test is unrelated to specific compliance rules, the development of compliance tests can focus on offering a highly reusable set of tests as a library for specifying compliance rules.
As there is a need to incorporate a software-centric point of view into compliance testing, we introduced Software Component, Component Landscape and System as model elements into the Continuous Compliance Model.
A Software Component represents a software artifact/product that has requirements regarding the configuration of the infrastructure and defines desired properties and aspects of a system or node. A software component can define variables to model properties which are required but can be assigned to context-specific values.
As an example of such a component conside Apache httpd. A corresponding variable could be the PORT for the web server software with a default value (e.g., PORT = 80).
A Component Landscape is a set of software components which form a working application or product. A component landscape can be distributed among different nodes in a network.
Software components and component landscapes can be associated with compliance rules and rules sets to model specific compliance requirements.
A Software System is the representation of a real operating software system running on various nodes in a network.
A software system is a specific set of component landscapes and components and comprises an Environment, which specifies the actual nodes in a network, where the components are running. Software components and landscapes can be part of multiple systems. On such a system a compliance test can be conducted, as it is accessible and assessable by compliance tests.
Based on these concepts, the necessary compliance tests to assess a system can be derived. Figure 3 depicts a concrete compliance testing scenario for the Apache httpd web server and a custom software component. These two components form a component landscape which is instantiated as a software system running on AWS. For Apache httpd the specified compliance rules and tests are added to the model.
To perform compliance testing in a continuous manner and to embed it into a continuous delivery process, we have to define possible interaction points, where a deployment pipeline can integrate and execute a compliance model. The software or framework operating on the continuous compliance model needs to manage multiple test executions to produce reports which can be evaluated by the pipeline.
To generate executable test cases, the compliance model which is represented as directed acyclic graph (DAG) is traversed. Each path represents an individual test case, which will be executed and evaluated. While traversing the model, Compliance Technical «Software Component» Application WS
«System» WebService@AWS «Environment»
AWS «Component Landscape»
WebService «Software Component» Apache httpd «Compliance Rule Set» SecureWebserver port=443 «Compliance Rule» Open Server-Port «Compliance Rule»
WebserverProtocol HTTP «Compliance Rule» Active SSL
«Compliance Rule» Valid Certificate «Compliance Test» TestForOpenPort «Compliance Test» TestForHTTP «Compliance Test» TestSSLConn «Compliance Test»
TestValidCert «Parameter»
Port variables are resolved top-down to their final corresponding value. At the technical level, all variable values are injected as parameters for the compliance tests, which are implemented as parameterized tests. So each path of the graph results in a Generated Test Case.
A Test Execution Report is an artifact containing all test results from all Test Executions, which ran a specific instance of the Continuous Compliance Model. To ensure traceability a test execution includes the executed generated test cases.
The example compliance model depicted in Figure 3 leads to the generation of four different test cases. The variable PORT defined on the software level will be delegated down to the technical level and injected as a parameter for each test.
1) Validation & Simulation: Based on a complete model validation and simulation activities can be performed to gain insights about the current compliance status without even running the tests explicitly, e.g. missing or conflicting variable definitions can be discovered on model level by analyzing different paths and the potential test cases. Contradictory rules can be discovered when modeling a new software landscape or system. Techniques like constraint solving can be applied to find a compliant configuration for the modeled system.
2) Risk computation: To each compliance rule a risk value can be assigned, which indicates the potential risk if it is not fulfilled by the SUT. All risk values will be aggregated along the paths and added in the root node, the system element, to gain a singular risk value for this specific software in this specific environment at the time of execution of the compliance tests.
3) Reusability: Our approach supports and enforces reuse of technical, compliance, and software model elements when modeling and adding new elements. Current compliance standards like CIS or BSI can be easily modeled as compliance rules or compliance rule sets and then be assigned to certain software components which are operated in the organization. By means of variables, the rules can be tailored to specific environments and systems without touching the implemented standard itself. Test case implementations can be reused in different rules for different purposes. In the end the tests, rules, and components form a standardized organization-wide library of compliance related artifacts.
4) Continuous Delivery: Continuous delivery and its
technical implementation, the deployment pipeline, define various
stages where testing activities can be performed. As
continuous compliance testing is a form of nonfunctional testing [
But in addition the compliance rules and tests can be used
for checking test systems and environments used during a
pipeline run, to ensure a proper compliance state before
performing additional functional testing or to test deployment and
provisioning mechanisms. Chef Automate7 uses this approach
to test Infrastructure As Code [
5) DevOps: As the compliance model integrates
information about the desired target production environment and about
how the software is operated it also supports the DevOps
mindset [
By supporting and implementing ShiftLeft, developers get in touch with operational principles and constraints. In agile projects this will have an impact on software design and architecture.
In addition, former non-related activities like IT-Governance will be aligned with software development. Due to the decoupled structure, the compliance model can be defined by different stakeholders. Compliance rules may be defined by a CSO but the corresponding compliance tests are provided by developers. We skip the organizational and cultural issues of compliance modeling and testing as this is out of the scope of this paper.
COMET is a web-based continuous compliance testing framework, based on the continuous compliance model (see figure 2) The user of COMET can add compliance tests, rules and software components to build a continuous compliance testing scenario. The architecture of COMET follows the bounded contexts introduced in Section IV defining technical, compliance, software, operation and execution related areas. The user can define and modify entities of these areas independently. To build a complete compliance testing model the user needs to provide or reuse elements of all areas.
COMET is able to execute a continuous compliance model, by traversing the model, resolving the variables with the system, software, or rule specific input-data, injecting these data into the parametrized compliance tests, then generates and executes the generated test cases. In the end the software will provide a report with the results of the execution.
Figure 4 depicts the COMET report screen after the execution. Sensitive informations are defaced by blurring these parts in the screenshot. All results, failed and successful tests, can be analyzed by the user. The system reports the accumulated risk value derived from all failed compliance tests. Currently, COMET supports only InSpec as possible test technologies, but the architecture provides a defined hot-spot to add new testing technologies.
The area of compliance and compliance testing has been
in the focus of research and industry for quite some time.
Frameworks like COBIT [
From a technical perspective some tools has been developed to perform compliance tests on a running system. But, the research topic “compliance as code” and continuous compliance testing are relatively new and not yet that present. Tools like Chef InSpec8, RedHat OpenScap9, UpGuard10, CISCAT11 and others allow to define and execute compliance tests. Although, the possibilities to automate compliance testing, to integrate it into continuous delivery and to manage multiple complex environments are limited. A software-centric point of view is not incorporated in any of these tools. But, COMET builds upon these technologies and uses them to implement compliance rules.
Academia and industry are discussing about extending
the scope from DevOps to SecDevOps, DevSecOps or
DevOpsSec [
Compliance testing is getting more and more attention as for example misconfigured infrastructures are one of the top security risks for software systems. A lot of effort has been spent by organizations like NIST or the BSI to collect knowledge and best practices to come up with guidelines and benchmarks. But these are quite general and focus on the infrastructure level. So, an organization operating or developing software struggles with ensuring compliance and at the same time incorporating all the necessities of their various software products.
This paper presents a novel approach to continuous compliance testing, which enables and provides a structured way to specify, organize and execute compliance tests. The approach focuses on integrating a software-centric view into compliance modeling. Therefore, it becomes necessary to model the actual software components in addition to the infrastructure and their relation to compliance rules. Compliance tests are derived by analyzing the specified software elements and their assigned compliance rules. The tailoring of these tests to specific environments is supported by using concepts of data-driven testing with the extension that the data is generated on demand based on the compliance model during execution. This eases the reuse of compliance rules as these are similar to parameterized
9https://www.open-scap.org/ 10https://www.upguard.com/ 11https://learn.cisecurity.org/cis-cat-landing-page 12https://docs.hashicorp.com/sentinel/ tests. Due to its generative nature, the approach can be easily automated and embedded into a continuous delivery process and so is paving the way to continuous compliance testing. Applying compliance testing in early stages of the deployment pipeline, by using the rules in integration test environments, supports fast and early feedback of the compatibility of the target environment with the most recent software versions.
We offered a first glimpse on our tool COMET, which implements our approach. A conducted case study, which uses COMET in an industrial context is out of scope of this paper.
In future we plan to further evaluate our approach and gain more insights into the challenges of compliance testing. Especially the integration of different stakeholders like developers, security experts, product owners and managers into the process of developing and maintaining continuous compliance models is challenging. We seek to integrate concepts of business IT alignment to tackle this challenge.