=Paper=
{{Paper
|id=Vol-2017/paper03
|storemode=property
|title=Static and Dynamic Architecture Conformance Checking: A Systematic, Case Study-Based Analysis on Tradeoffs and Synergies
|pdfUrl=https://ceur-ws.org/Vol-2017/paper03.pdf
|volume=Vol-2017
|authors=Jan Thomas,Ana Nicolaescu,Horst Lichter
|dblpUrl=https://dblp.org/rec/conf/apsec/ThomasNL17
}}
==Static and Dynamic Architecture Conformance Checking: A Systematic, Case Study-Based Analysis on Tradeoffs and Synergies==
https://ceur-ws.org/Vol-2017/paper03.pdf
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
Static and Dynamic Architecture Conformance
Checking: A Systematic, Case Study-Based
Analysis on Tradeoffs and Synergies
Jan Thomas Ana Nicolaescu Horst Lichter
RWTH Aachen University RWTH Aachen University RWTH Aachen University
jan.thomas@rwth-aachen.de ana.nicolaescu@swc.rwth-aachen.de horst.lichter@swc.rwth-aachen.de
Abstract—In order to uncover architectural drift, a plethora In addition, with the advent of new architecture styles (e.g.
of architecture conformance checking tools has been proposed microservices), the complexity of software systems shifts from
that mainly leverage two approaches: they extract architec- their static structure to their run-time behavior. Consequently,
tural knowledge based on either source code artifacts (static
approach) or run-time behavior (dynamic approach). Although static analysis tools might not cover all architecturally relevant
both approaches have been evaluated separately, no up-to-date aspects. To address this issue, tools were proposed [13] [14]
analysis of their relative strengths and weaknesses, nor real- [15] that collect architectural evidence by means of run-time
world comparative case studies of the two were published. In data and therefore cover behavioral aspects as well (dynamic
this paper we address this issue by presenting the results of approach).
a direct comparison of both approaches. We first identify and
compare their strengths and weaknesses on a theoretical level. Although both approaches have been discussed separately,
We then evaluate these results against our experiences gained in no work addressed a direct comparison thereof. Guided by this
a large-scale industrial case study. As a result, we argue that the observation, we derived two research questions:
approaches cannot substitute each other as they differ in many
key aspects. Hence, we crystallize guidelines regarding how to • RQ1 - Considering their general capabilities, which are
combine these such that their strengths are emphasized while the main strengths and weaknesses of static and dynamic
weaknesses mitigated. architecture conformance checking approaches?
• RQ2 - How can both approaches be combined in order
I. I NTRODUCTION to obtain synergies?
It is well studied and understood [1] [2] that the quality Addressing these questions on a theoretical foundation can
of a software system’s architecture directly influences its non- be done by consulting existing literature and related work.
functional properties (NFP) like understandability, maintain- However, answering such questions outside the scope of a
ability, or security. As a consequence, developing successful real-world context is not recommended, as results may lack
large-scale software solutions requires a well-documented ar- applicability and industrial relevance. Hence, we also con-
chitecture. If the documented architecture accurately reflects ducted a case study as a six months long project that emerged
the built system, it serves as the key artifact for architectural within the cooperation with one of our industry partners.
communication among different stakeholders and sound ar- During this case study, we explored the applicability of both
chitectural decision making [3]. However, as software sys- architecture conformance checking approaches in a large-scale
tems evolve, it is often the case that a gap between the industrial context. As a result, the key contributions of this
actually implemented architecture (as-implemented) and its paper are bivalent. (1) We contribute a theoretical comparison
documentation (as-intended) emerges. This gap is referred to of static and dynamic approaches in general and (2) undergo
as architectural drift [4]. If this drift is not addressed properly, a comparison based on the experiences gathered from our
it could cause the violation of architectural design decisions industrial case study. Both are important contributions towards
(architectural erosion) [5], which can cause serious problems answering the research questions as defined before.
related to the NFPs listed above. The remainder of this paper is structured as follows: Sec-
In order to uncover architectural drift, several architecture tion II investigates the theoretical background of static and
conformance checking techniques were proposed [6] (e.g. dynamic approaches. Subsequently, we introduce the context
reflexion modeling [7]) and implemented in commercial [8] and design of our case study in Section III. Next, we present
[9] and non-commercial [10] tools. These tools collect archi- the challenges that we encountered while performing the case
tectural evidence from a system’s source code and compare it study and continue with the evaluation of the obtained results.
to an as-intended architecture (static approach). However, cur- A thorough comparison and discussion of both approaches
rent research shows that automated architecture conformance based on the previous results is subject to Section IV. Section
checking techniques and tools are still not well adopted by the V presents related work. Last, Section VI concludes the paper
industry [11] [12]. with a summary and a discussion of future work.
Copyright © 2017 for this paper by its authors. 6
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
II. BACKGROUND OF S TATIC AND DYNAMIC A PPROACHES TABLE I
R ELATIONS OBTAINED FROM ARCHITECTURAL EVIDENCE
In comparison to other techniques for checking architecture
conformance, reflexion modeling has a large industrial accep- Relation Type Static Appr. Dynamic Appr.
Import x
tance and a great maturity of tool support [16]. Originally, re-
Extends x
flexion modeling was introduced [7] as a technique to uncover Implements x
architectural drift based on three prerequisites: an as-intended Variable Access x x
architecture model, low-level architectural evidence and a Instantiation x x
Method Invocation x x
mapping between both. Based on these inputs, entities and Dynamic Method Invocation x
relations of the as-intended architecture model are identified Inter-Process Communication x
as architectural convergences, divergences or absences. The Execution Frequency x
capabilities of reflexion modeling strongly depend on the type Execution Time x
of architectural evidence (entities and relations) that can be
extracted by either analyzing source code (static approach)
evidence which can be obtained and how it is extracted, we
or by analyzing run-time data from an instrumented system
can derive the strengths and weaknesses of either approaches.
(dynamic approach).
Using reflexion modeling techniques, the as-intended archi-
A. Architectural Entities and Relations tecture can be expressed as rules of the type “is allowed to
We illustrate which entities and relations can be obtained use” (e.g. component A is allowed to use component B).
for both approaches on the example of Java. Analyzing Java The type of usage can be refined based on the extracted
source code, one can obtain three kinds of entities: files, architectural relation types. Compared to some basic usage
packages and types (classes, enums or interfaces). Each source rules, which can be formulated in both approaches, more
file defines a package that it belongs to and defines a set of im- complex ones can be defined based on the relations obtained
ported types. Classes can extend other classes and implement by the dynamic approach. First, it can distinguish multiple
interfaces. Interfaces are able to extend other interfaces. Within method invocation types (e.g. direct invocation, database ac-
classes, a different set of relations can be found. Classes can be cess, remote procedure calls, web services, message bus).
instantiated, variables of classes or instances can be accessed Second, communication parameters captured at run-time (e.g.
and methods of classes or instances can be invoked. While the method invocation arguments or web service endpoint) can
belongs to package relation defines the structure of entities, also be considered when modeling usage-based rules. For
all remaining ones (import, extend, implement, instantiate, instance, architects could define rules that restrict access to
variable access, method invocation) express different kinds a component through a specific REST API. Similarly, given
of usage. As it is possible to define these relations during a component, the architects can choose to restrict its database
run-time1 , it is possible that the resulting relations are not access to a specific table. Third, besides is allowed to use
created ahead of run-time. Hence, usage that is triggered in this rules, the dynamic approach also facilitates the definition of
dynamic manner is not within the scope of the static approach. rules based on the temporal order of captured method calls
In contrast, applying the dynamic approach one needs to or their frequency. For instance, one could define rules stating
extract architectural evidence through instrumentation of a that a method call A must happen before B or that A must
system using special monitoring tools like Dynatrace [17] not be called more than once. Generally speaking, the dy-
and triggering its behavior (e.g. running test-cases), which namic approach facilitates the analysis of richer architectural
we refer to as episodes. The monitoring tools intercept and properties.
capture method invocations at run-time and yield a set of B. Distinguishing Capabilities
so-called execution traces. A trace can be defined as a tree
of caller-callee-nodes. Each node includes the name of the A key benefit of the dynamic approach is the ability to
invoked method and the fully qualified name of the type analyze inter-process communication. This becomes essen-
it belongs to. Furthermore, nodes can be augmented with tial when analyzing systems following new architecture styles
additional run-time information (e.g. object ids or method like microservices. The complexity of these systems shifts
arguments). In addition, the execution frequency of a method from their static structure to their interaction and behavior at
can be derived from all observed invocations. Dependent on run-time, as they are distributed across multiple processes. In
the utilized monitoring tool, further relations are in the scope these cases, only the dynamic approach can provide insights
of the dynamic approach. This includes method invocations about the entire system, as the concrete architecture is only
which are triggered dynamically (e.g. by reflection) as well assembled at run-time.
as variable access. Furthermore, relations resulting from inter- Even if architectural conformance analysis is restricted to
process communication (e.g. pipes, database access, or web homogeneous single process systems, not all architecturally
services) can also be extracted. Table I summarizes the previ- relevant information can be obtained from the source code
ous analyzed architectural relations. Based on the architectural [3]. For instance, relations caused by late binding (e.g.
polymorphism) can only be extracted from running systems.
1 using techniques such as reflection and dependency injection However, late binding effects might not always be desired.
Copyright © 2017 for this paper by its authors. 7
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
TABLE II
S YSTEM UNDER ANALYSIS (SUA) - S IZE - ORIENTED METRICS
Metric Value
Lines of code ∼ 125000
Number of Apache Maven projects 51
Number of OSGi bundles 30
Number of packages 138
Number of source files 879
Fig. 1. Reflexion models - Application on the dynamic approach
In order to make a reasonable episode selection, one could
For instance, interfaces are often used as a measure to prevent follow guidelines to ensure that these cover the system’s most
direct coupling between classes. While the static approach is relevant use-cases. Besides using such guidelines, the quality
aware of this decoupling, the dynamic approach is not because of selected episodes should also be measured objectively. To
it observes the dynamic, late bound type at run-time. This this end, architects can adopt structural coverage metrics (e.g.
anomaly needs to be considered by architects when modeling statement coverage) in the context of architectural confor-
rules for either the static or dynamic approach. mance checking to assess the quality of selected episodes
A major difference between the static and dynamic ap- (see e.g. [19] or [20]). Measuring coverage metrics bears
proach is the analysis completeness [18]. While architectural the advantage of gaining confidence regarding the analysis
evidence extracted from source code holds for all program significance. In addition, these metrics are easy to measure
executions, architectural evidence obtained from run-time data due to mature tool support for major programming languages.
only holds for the observed behavior. As a consequence, the To sum up, in this section we defined the analysis scope
dynamic approach cannot guarantee that a system satisfies a of both approaches based on entities and relations obtained
particular architectural property, but can only detect violations from architectural evidence. In addition, we distinguished the
of those. In contrast, the static approach gives this guarantee, approaches by capabilities of analyzing interprocess com-
as its properties hold for all program executions. In case munication, late binding effects and characteristics regarding
that both approaches disagree regarding a certain property, completeness.
either the dynamic approach did not cover all relevant parts
of the system or the static approach analyzed code that is III. C ASE S TUDY
either unreachable or not used anymore [18]. As the dynamic This section describes the conducted case study. We first
approach considers actual program executions, it does not describe the case study context and design, before we comment
suffer from analyzing unreachable or unused code. However, on encountered challenges and present the obtained results.
due to the same reason it is hard to achieve completeness.
Applying these thoughts regarding completeness to reflexion A. System Under Analysis (SUA)
modeling techniques, adds a fuzziness to the results of the Subject to this case study is a system for task automation
dynamic approach (cf. Figure 1). Parts of the as-implemented and data distribution, which is developed since 2008. As part
architecture that were not captured due to incomplete instru- of the software evolution process and driven by customer
mentation are represented as dashed rectangles. Taking the needs, features were constantly added and improved. As a
possibility of uncovered parts into account leads to three consequence, its architecture was refactored several times.
considerations. First, if not all parts of the as-implemented Today, the Java OSGi based system comprises a total of
architecture are covered, it is not certain that the set of diver- 125000 lines of code. Table II summarizes additional size-
gences reflect all actual divergences in the software system. oriented metrics. As the SUA represents a business critical
There might be architectural violations in the system which are component, it is crucial to ensure that all non-functional
not detected, thus this set only suites as a lower bound. Second, properties considered by the architects are actually respected
absences may be detected that actually are convergences due to in the implementation. Hence, there is a strong need for
uncovered parts in the as-implemented architecture. Therefore, automated architectural conformance checks.
the set of detected absences can only be interpreted as an upper The system is divided into five processes, each implement-
bound, whereas the set of detected convergences represents a ing an individual functional slice. Each process runs an OSGi
lower bound. Third, the probability that these match the actual container which hosts several OSGi bundles. These in turn can
sets in the software system is influenced by the degree of provide or require OSGi services and depend on other .jar
uncovered parts in the as-implemented architecture. If a large artifacts. Persistence is realized through a RDBMS and local
uncovered fraction exists, this probability is low. In contrast, XML files. Inter-process communication (IPC) among the five
lowering this fraction increases the robustness and reliability processes is implemented by polling for state changes in the
of the results. As a consequence, architects need to make a persistence layer as well as by publishing and subscribing
reasonable episode selection such that all parts of the system to a message bus. As data distribution is a central feature
that are relevant for the analysis are covered. of the SUA, it implements several communication protocols
Copyright © 2017 for this paper by its authors. 8
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
TABLE III the SUA by using ARAMIS’ dedicated architecture descrip-
SUA - A RCHITECTURE DIAGRAMS tion language (ADL). This description includes a hierarchy
of so-called architecture units, their mapping to the source
Diagram Concepts Dependencies
code elements and their allowed interaction rules. Second,
System Processes, External Protocols,
Message Bus File System
the SUA needs to be instrumented while performing certain
episodes in order to capture low-level architectural evidence
Process OSGi Bundles External Protocols,
File System
in form of run-time traces. As ARAMIS builds on top of well
established monitoring infrastructures, we used the Dynatrace
OSGi Bundle External Protocols,
File System, .jar artifacts, [17] monitoring infrastructure, as it is able to monitor complex
OSGi Services distributed and heterogeneous systems. Third, the captured
traces need to be analyzed by the ARAMIS processing chain
regarding architectural violations.
(e.g. FTP, HTTP, etc.). Its architecture is documented in a set
of UML component diagrams which depict several levels of D. Case Study Design and Execution
abstractions: system, process and OSGi bundle. Each diagram During an initialization phase, we organized several meet-
models how the inner concepts are connected and defines ings to discuss the general setting and high-level requirements
dependencies to the context in which it is embedded. Table for automated architecture conformance checks. Afterward, the
III depicts which concepts and dependencies are represented source code (as-implemented architecture) and the diagrams
in which diagram type. (reference as-intended architecture) were manually inspected.
Based on these diagrams and expert interviews, we derived Based on this inspection and expert interviews, architectural
properties that had to be analyzed in the case study: (1) Do the questions of interest tailored to the SUA were derived. As the
OSGi bundles use just the documented .jar dependencies? (2) utilized tools can not directly perform conformance checks
Are OSGi services wired at run-time as documented? (3) Is the based on the reference model, tool-specific as-intended archi-
database used only through the access layer component? (4) tecture models needed to be created. We decided to exercise
Are only the allowed components coupled at run-time through an automated model-to-model transformation approach which
the message bus? (5) Are only white-listed directories and files was studied in our previous work [21]. This approach en-
accessed at run-time? (6) Are external systems always used abled us to obtain tool-specific models of comparable quality,
through their dedicated protocol facades? While properties 1, which facilitates a fair comparison of the static and dynamic
3 and 6 are related to the static structure of the SUA, the approach. Another prerequisite for applying the tools is the
analysis of property 2, 4 and 5 requires insights into the availability of architectural evidence. While the source code
system’s run-time. For this reason, we decided to combine could be accessed easily by Sonargraph-Architect, we had
the tools Sonargraph-Architect (static approach) and ARAMIS to instrument the SUA and capture traces at run-time for
(dynamic approach) for this case study. ARAMIS using Dynatrace. To this end, we used an already
existing UI test suite and measured its statement coverage
B. Employed Static Tool - Sonargraph-Architect to assess its adequacy for supporting a behavior-based con-
Sonargraph-Architect [8] is a well established tool for static formance check. Using the automatically transformed models
code analysis. It allows to monitor complex software systems and the architectural evidence, we then applied the tools
regarding their technical quality and to enforce architectural Sonargraph-Architect and ARAMIS on the SUA in order
conformance rules. In addition, it calculates a variety of to uncover its architectural drift. Subsequently, we manually
metrics. While Sonargraph supports many features, we just verified our results by classifying the detected architectural vi-
used the architecture conformance related features for this olations into defects in the as-intended architecture, defects in
case study. Sonargraph-Architect provides a domain-specific the as-implemented architecture and false positive violations.
language (DSL) to define a system’s architecture. According This activity was conducted in cooperation with architects and
to the principles of reflexion modeling, the DSL supports the developers.
definition of components which are mapped to source code and
connected by architectural rules (e.g. allowed to instantiate, E. Challenges
allowed to call, etc.). This section illustrates challenges we encountered when
conducting the case study and explains countermeasures we
C. Employed Dynamic Tool - ARAMIS took.
ARAMIS (Architecture Analysis and Monitoring Infrastruc- As stated above, we followed a model-to-model transforma-
ture) is a tool-supported framework for run-time monitoring, tion approach to obtain tool-specific as-intended architecture
communication integrity validation, evaluation and visualiza- models from a set of reference as-intended architecture dia-
tion of the behavior view of software architectures [15]. In grams. When implementing this transformation, we encoun-
order to automatically check for architectural conformance tered two major challenges. First, we needed to automatically
using ARAMIS, three steps need to be followed. First, an map components in the reference model to implemented
as-intended architecture description needs to be defined for classes or packages in the source code. Second, the diagrams
Copyright © 2017 for this paper by its authors. 9
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
denoted connections between components in terms of Java analyzing the execution frequency captured at run-time. These
interfaces. While the static approach is aware of interface valuable run-time insights highlight the unique capabilities of
decoupling, the dynamic approach is not due to late binding. the dynamic approach.
As traces captured at run-time do not state usage of an Although we monitored about 140 automated UI test cases,
interface, but usage of a concrete implementation, rules in the only 32% of all statements in the SUA were covered at run-
dynamic approach need to be aware of the concrete implemen- time. While this low coverage does not give any confidence
tations. However, as these were not denoted in the reference that we captured a relevant part of the SUA, the detailed
diagrams, it was not possible to automatically generate these coverage report gives valuable hints how the utilized test suite
rules without any additional information. The presented issues could be improved (e.g., only 26% of the system’s public API
boil down to two questions: (1) Which packages and classes were covered).
are defined within a component given its name? (2) Which Once set up, an important performance indicator of archi-
classes implement a particular interface given its name? In tecture conformance checks is the cycle time for the whole
order to automatically answer these questions, we decided to analysis process. We experienced cycle times of less than
embed a static pre-analysis of the component’s source code a minute for the static approach. In contrast, the dynamic
into the transformation process. This analysis creates two approach was a long running process which took 13 hours
mappings by iterating over all components and their classes: (deduplicated traces) up to 33 hours (full traces).
(1) component name ⇒ included packages and classes names,
(2) interface name ⇒ implementations’ class names. By using IV. C OMPARISON AND D ISCUSSION
these mappings, the previous questions could automatically be In the previous sections we mainly elaborated on the basic
answered, which enabled us to apply the planned model-to- characteristics of both approaches (RQ1). In contrast, this
model transformation approach. section first consolidates the previous results for a comparison
As usual with dynamic approaches, scalability issues needed of both approaches. Subsequently, we discuss how both ap-
to be carefully addressed. The monitoring of the approximately proaches can be combined in order to obtain synergies (RQ2).
140 test cases lasted for 33 hours and produced 16 GB of trace
information comprising over 35 million caller-callee-pairs. To A. Comparison
reduce the amount of data, we applied our knowledge regard- Knodel and Popescu derived different comparison dimen-
ing the monitored tests and the system’s general architecture: sions as part of their work [6] in order to compare three
redundant traces result through (1) test fixture setup and tear different static architecture conformance checking approaches.
down and (2) periodic polling employed by the processes. We Being generic dimensions, we reuse a subset of these and
discarded these duplicate traces by utilizing heuristic-based adapt them for our work. In particular, these dimensions cover
data deduplication techniques. This lead to a comprehensible required inputs, involved stakeholders, manual tasks, the anal-
reduction of over 60% of the data to be analyzed and its ysis scope, the analysis completeness, evaluation performance,
processing time. scalability factors and maintenance aspects. The following
discusses our comparison results which are summarized in
F. Results Table IV.
When exercising both approaches, we obtained 20 types Before choosing a conformance checking approach, com-
of architectural violations using ARAMIS and 15 violations panies need to be aware of prerequisites in terms of required
using Sonargraph-Architect. Only 3 of these violation types inputs . Both approaches require a tool-specific as-intended
were detected by both tools. With the help of architects, architecture model which can either be modeled manually
we further classified these into defects in the as-intended or be transformed automatically from a reference model. In
architecture, defects in the as-implemented architecture and our case study, we augmented the reference model by static
false positive violations. About two fifth were identified as information gathered from source code. Furthermore, both
false positives due to false mapped classes or due to false conformance checking approaches depend on architectural
modeled rules. The majority of the remaining violations was evidence for their analysis. The static approach can infer
traced back to imprecise information in the reference as- this evidence directly from source code. In contrast, the
intended architecture model. Sonargraph-Architect uncovered dynamic approach needs to capture run-time traces from an
undocumented external dependencies, which were not in the instrumented system for this task. In addition, this process
scope of ARAMIS as we reduced its instrumentation scope. In requires a solid instrumentation configuration and an episode
addition, Sonargraph-Architect detected 4 architectural viola- selection that captures all architecturally relevant parts of the
tions, which were not detected by ARAMIS due to insufficient SUA. These inputs need to be available upfront the analysis
coverage produced by the monitored test suite. Both observa- and maintained throughout the systems lifetime by different
tions confirm our previous thoughts on completeness of the stakeholders.
dynamic approach. In contrast, ARAMIS was able to uncover For both approaches, the creation of tool-specific architec-
file system related violations which were not in the scope ture models involves manual work either by manually creating
of Sonargraph-Architect. In addition, we identified an unused the model or by implementing a tailored model-to-model
database connection and two performance critical violations by transformation. However, if a similar system was analyzed
Copyright © 2017 for this paper by its authors. 10
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
TABLE IV
C OMPARISON OF THE DYNAMIC AND STATIC APPROACH
Approach
Dynamic Static
Dimension
Inputs • as-intended architecture model, source code • as-intended architecture model, source code
• episode selection, instrumentation configuration
Manual Tasks • create initial as-intended architecture model • create initial as-intended architecture model
• review results • review results
• orchestration of process steps
Analysis Scope • heterogeneous systems • homogeneous systems
• inter-process communication, late binding • static structure
Completeness • not complete • complete
• approximately characterized by coverage metrics
Evaluation Performance • long running process • instant feedback
Scalability Factors • instrumentation overhead • none experienced
• resource usage increases with captured traces
Maintenance Subjects • as-intended architecture models • as-intended architecture models
• instrumented system, episode selection
before, the transformation might be reused or adapted. Beyond proach is the analysis completeness. The static approach is
that, the static approach does not need much manual effort complete within its scope as it can access all architectural
to carry out the analysis process due to mature tool support. relations easily from source code. In contrast, the dynamic
In contrast, behavior-based non-commercial tools need much approach is not complete because in practice it is not possible
more manual effort as they are not as mature. In the case of to capture behavior of an application in its entirety. As a conse-
ARAMIS, trace capturing, importing and processing need to quence, the dynamic approach cannot guarantee that a system
be initiated and monitored manually by architects. Another satisfies a particular property for all program executions. As
important manual task is the discussion of detected violations stated before, the degree of completeness depends on the
by architects and developers in order to identify their cause and quality of selected episodes. However, increasing the quality
possible solutions. As architectural violations can be caused of these, such that they cover more parts of the SUA, has two
individually by defects in the as-intended architecture, by major influences. First, it requires significantly more effort to
defects in the as-implemented architecture or by defects in the write and maintain test suites for those episodes. Second, as
analysis process (false positives), this task can not be handled more episodes are monitored, more traces are captured. As
by tools. a consequence, the overall analysis has an increase in trace
capturing time, processing time and storage requirements.
In order to choose the right approach, it is important to
consider the approachs analysis scope. As identified before,
static tools collect architectural evidence directly from source Both approaches differ significantly in their cycle time of
code. Hence, the scope of a particular analysis is constrained the overall analysis process. In the context of our case study,
to a single programming language and in turn to homoge- we experienced short cycle times of less than a minute for the
neous systems only. In contrast, the analysis scope of the static approach and long cycle times of 13 to 33 hours for the
dynamic approach mainly depends on the utilized monitoring dynamic one. Therefore, the static one offers the opportunity
infrastructure. Advanced ones (e.g. Dynatrace) are able to to provide instant feedback, whereas the dynamic one is a long
collect traces across system boundaries. Therefore, these tools running process which needs to be planned and scheduled in
facilitate the analysis of heterogeneous systems-of-systems. advance.
Architectural relations that can be analyzed by a particular
approaches were illustrated in Table I. Summarizing this table,
Finally, one needs to take maintenance aspects into ac-
the static approach has to be used if the analysis should include
count, before applying one or the other approach. For both
rules based on import, extends or implements relations. On the
approaches it is crucial to maintain the reference and tool-
contrary, the dynamic approach has to be applied if dynamic
specific as-intended architecture for the SUA. Two additional
method invocation, inter-process communication, execution
artifacts need to be maintained for the dynamic approach. First,
frequency or timing related aspects are relevant. Rules based
an instrumented version of the SUA needs to be maintained
on method invocation can be modeled in either approaches.
in order to capture traces. As the source code changes, its
However, if the dynamic type of an invoked method (late
instrumentation configuration must be updated accordingly.
binding) is relevant for the analysis, the dynamic one needs to
Second, the episode selection must be maintained accordingly
be used as the static one is not aware of late binding concepts.
to changes in the source code or the architecturally relevant
A pivotal difference between the static and dynamic ap- use cases of the system.
Copyright © 2017 for this paper by its authors. 11
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
B. Discussion of existing tools and techniques. Static approaches were dis-
The previous comparison points out that both approaches cussed in [6] [23] [10]. Approaches focusing on the behavior
have their own strengths, weaknesses and unique features. of software systems were studied in [13] [14] [15]. But, to
Hence, one approach can not substitute the other. However, the the best of our knowledge, there is no work that performed a
dynamic one requires significantly more effort and resources direct comparison of both approaches, based on a theoretical
compared to the static one in almost all dimensions. As level and case study results.
a result, the static approach should be preferred according Knodel and Popescu [6] conducted a comparison of three
to the “least effort extraction” strategy [3] for analyzing low-level static architecture conformance checking techniques
architectural properties which are within the scope of both in the context of a tool for static architecture evaluation
approaches. Beyond that, we propose to first establish a solid called SAVE. Guided by a goal-question-metric approach,
static architecture conformance analysis before applying the they derived 13 dimensions in order to compare the three
dynamic approach. According to the pareto principle, a static techniques. We used a subset of these dimensions and adopted
analysis can already identify a large fraction of architectural them for our work in section IV. Like in our work, the authors
violations using comparatively little effort. In addition, it bears identified strength and weaknesses for each technique. They
the advantage of being complete within its scope. Due to proposed that architects should individually choose the right
the ability of providing instant feedback, the static approach technique for their needs based on the comparison dimensions
could potentially be integrated into IDEs, which has two and results.
advantages. First, it increases the architectural awareness and In [24], Knodel et al. reported a long term experience on
hence prevents the emergence of severe architectural violations transferring static architecture conformance checking to their
at development time. Second, if it is applied frequently, industry partner. They identified the need for automating the
it facilitates a tight feedback loop between developers and process to a large extent due to time constraints of their
architects by continuously validating architectural decisions. industry partner. In [25] Rosik et al. present their results
In summary, the static approach should be used early and often obtained while applying static architecture reconstruction over
in order to establish the foundation of tool-based architecture a time-span of two years for one of their industrial partners.
conformance checking. Among others, they recommend to apply an adaptation of the
reflexion modeling technique: conformance checking should
However, using solely static tools is not sufficient for a
be undergone periodically during the actual development and
comprehensive architecture analysis. Based on our analysis
not only on the completed system. Similarly, Ganesan, Keuler,
and the performed case study, we see the following two
and Nishimura [20] share their experience applying the dy-
scenarios where the dynamic approach can enhance the results
namic approach in an industrial context. They experienced
of a static analysis
that conformance checking is best applied iteratively and in
First, if analyzing heterogeneous systems-of-systems, a dy-
close cooperation with architects. Furthermore, they propose to
namic analysis focusing on inter-process communication can
measure the code coverage metrics of monitored use cases to
augment the static analysis, which focuses on inner-process
ensure that all relevant architectural components were covered
properties only; in particular, a static analysis needs to be aug-
at run-time. This approach was also proposed by [19] and [26].
mented by a dynamic one, if architectural relevant properties,
Similar to these case studies, we also provided architecture
like security or performance are influenced by concepts such
conformance checks as a service. In addition, we automated
as late binding, etc..
the creation of tool-specific architecture models and captured
Second, as applying just the static approach does not give
coverage metrics to assess the quality of our dynamic analysis.
insights how often particular violations occur at run-time, the
In contrast to these studies, we applied the static and dynamic
dynamic approach can be used to measure the frequency of
approach in parallel. This facilitated a comprehensive compar-
architectural violations in order to assess their severity.
ison of both approaches and a broader architectural analysis
To sum up, we propose the combination of static an
scope.
dynamic approaches in order to achieve a broader analysis
scope. The static approach should be preferred for all archi- VI. C ONCLUSION AND F UTURE W ORK
tectural properties where it is applicable as it is complete
In this paper, we studied the characteristics, strengths and
and requires less effort. As it provides instant feedback, it
weaknesses of static and dynamic architecture conformance
should be used frequently by developers in order to increase
checking approaches and examined how synergies can be
architectural awareness and prevent architectural drift. To com-
obtained by combining both. In the following, we draw a
plement missing architectural properties of the static analysis,
conclusion regarding our initial research questions.
a dynamic one should be conducted on demand.
RQ1 - Considering their general capabilities, which are the
main strengths and weaknesses of static and dynamic archi-
V. R ELATED W ORK
tecture conformance checking approaches? Static and dynamic
Architecture conformance checking has been in the focus of approaches are contrary to each other as most strengths of one
research for a long time. Ducasse and Pollet [22] introduced a approach are the weaknesses of the other. We studied this on a
taxonomy of the field and presented a comprehensive overview theoretical foundation (cf. section II) and supported our results
Copyright © 2017 for this paper by its authors. 12
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
with an industrial case study (cf. section III). On the one hand, [7] G. C. Murphy, D. Notkin, and K. Sullivan, “Software Reflexion Models:
the static approach enables an architectural analysis which is Bridging the Gap Between Source and High-level Models,” ACM
SIGSOFT Software Engineering Notes, vol. 20, no. 4, pp. 18–28, oct
complete within its scope spending comparable little effort. In 1995.
addition, it provides instant feedback and therefore can be used [8] “hello2morrow - Sonargraph Architect.” [Online]. Available:
frequently. On the other hand, its scope is limited to the static https://www.hello2morrow.com/products/sonargraph/architect9
[9] “Structure101 Software Architecture Development Environment
structure of software systems, which excludes the analysis of (ADE).” [Online]. Available: http://structure101.com/
inter-process communication, dynamic method invocation and [10] L. J. Pruijt, C. Köppe, J. M. van der Werf, and S. Brinkkemper,
effects of late binding. However, these concepts are within “HUSACCT: Architecture Compliance Checking with Rich Sets of Mod-
ule and Rule Types,” in Proceedings of the 29th ACM/IEEE international
the scope of the dynamic approach. But in contrast to the conference on Automated software engineering (ASE). ACM Press,
static one, it is not complete and requires significantly more 2014, pp. 851–854.
effort and processing time. In turn, it is aware of the frequency [11] A. Caracciolo, M. F. Lungu, and O. Nierstrasz, “How Do Software
Architects Specify and Validate Quality Requirements?” Springer,
violations occurred at run-time, which can be used to assess Cham, 2014, pp. 374–389.
their severity. [12] I. Melo, G. Santos, D. D. Serey, and M. T. Valente, “Perceptions of
RQ2 - How can both approaches be combined in order to 395 Developers on Software Architecture’s Documentation and Con-
formance,” in 2016 X Brazilian Symposium on Software Components,
obtain synergies? As both approaches are contrary to each Architectures and Reuse (SBCARS). IEEE, sep 2016, pp. 81–90.
other, one can not substitute the other. In order to achieve a [13] D. Garlan, S. Cheng, A. Huang, B. Schmerl, and P. Steenkiste, “Rain-
broader analysis scope, we proposed a way of combining both bow: Architecture-Based Self-Adaptation with Reusable Infrastructure,”
Computer, vol. 37, no. 10, pp. 46–54, oct 2004.
such that their strengths are emphasized and weaknesses are [14] B. Schmerl, J. Aldrich, D. Garlan, R. Kazman, and Hong Yan, “Dis-
mitigated. To utilize the completeness, low effort and instant covering Architectures from Running Systems,” IEEE Transactions on
feedback of the static approach, it should be preferred over the Software Engineering, vol. 32, no. 7, pp. 454–466, jul 2006.
[15] A. Nicolaescu, H. Lichter, A. Göringer, P. Alexander, and D. Le,
dynamic one if it is applicable and should be used frequently. “The ARAMIS Workbench for Monitoring, Analysis and Visualization
If the static analysis does not cover all relevant properties and of Architectures based on Run-time Interactions,” in Proceedings of
the project resources permit, a dynamic analysis can be used to the 2015 European Conference on Software Architecture Workshops
(ECSAW). ACM Press, 2015, pp. 57:1–57:7.
cover the missing properties. As it is not complete, structural [16] S. Herold, M. English, J. Buckley, S. Counsell, and M. O. Cinneide,
coverage metrics should be measured in conjunction to the “Detection of Violation Causes in Reflexion Models,” in Proceedings
analysis in order to expose uncovered parts of the analyzed of the 22nd IEEE International Conference on Software Analysis,
Evolution, and Reengineering - SANER ’15. IEEE, mar 2015, pp.
system. Due to its high effort and processing time, dynamic 565–569.
architecture conformance checks should only be conducted on [17] “Dynatrace - Digital Performance & Application Performance
demand. Monitoring.” [Online]. Available: https://www.dynatrace.com/
[18] B. Thoms, “The concept of dynamic analysis,” ACM SIGSOFT Software
It needs to be stressed that our practical results rest upon Engineering Notes, vol. 24, no. 6, pp. 216–234, nov 1999.
the analysis of a single software system. As it is not possible [19] H. Yan, D. Garlan, B. Schmerl, J. Aldrich, and R. Kazman, “DiscoTect:
to generalize from a single case, additional cases in different A System for Discovering Architectures from Running Systems,” in Pro-
ceedings of the 26th International Conference on Software Engineering
industrial settings should be studied in the future to validate (ICSE). IEEE, 2004, pp. 470–479.
our results. Nevertheless, in comparison to our theoretical [20] D. Ganesan, T. Keuler, and Y. Nishimura, “Architecture Compliance
work the conducted case study bears the advantage that it is Checking at Runtime: An Industry Experience Report,” in Proceedings
of the Eighth International Conference on Quality Software (QSIC).
based on a real-life software system. IEEE, aug 2008, pp. 347–356.
[21] D. Le, A. Nicolaescu, and H. Lichter, “Adapting Heterogeneous ADLs
ACKNOWLEDGMENT for Software Architecture Reconstruction Tools,” in Proceedings of
the 10th International Conference on Software Engineering Advances
We want to thank our cooperation partners from KISTERS (ICSEA). Barcelona, Spain: IARIA XPS Press, 2015, pp. 52–55.
AG for enabling us to perform the industrial case study [22] S. Ducasse and D. Pollet, “Software Architecture Reconstruction: A
presented in this paper. Process-Oriented Taxonomy,” IEEE Transactions on Software Engineer-
ing, vol. 35, no. 4, pp. 573–591, 2009.
[23] L. Passos, R. Terra, M. T. Valente, R. Diniz, and N. das Chagas
R EFERENCES Mendonca, “Static Architecture-Conformance Checking: An Illustrative
[1] R. Kazman and L. Bass, “Toward deriving software architectures from Overview,” IEEE Software, vol. 27, no. 5, pp. 82–89, sep 2010.
quality attributes,” Defense Technical Informtion Center (DTIC), Tech. [24] J. Knodel, D. Muthig, U. Haury, and G. Meier, “Architecture Compli-
Rep., 1994. ance Checking - Experiences from Successful Technology Transfer to
[2] L. Chung and J. C. S. do Prado Leite, “On Non-Functional Requirements Industry,” in Proceedings of the 12th European Conference on Software
in Software Engineering.” Springer Berlin Heidelberg, 2009, pp. 363– Maintenance and Reengineering (CSMR). IEEE, apr 2008, pp. 43–52.
379. [25] J. Rosik, A. Le Gear, J. Buckley, and M. Ali Babar, “An Industrial
[3] L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice, Case Study of Architecture Conformance,” in Proceedings of the Second
2nd ed. Boston, MA, USA: Addison-Wesley Professional, 2003. ACM-IEEE International Symposium on Empirical Software Engineer-
[4] R. N. Taylor, N. Medvidovic, and E. M. Dashofy, Software Architecture: ing and Measurement (ESEM). ACM Press, 2008, pp. 80–89.
Foundations, Theory, and Practice. Wiley Publishing, 2009. [26] L. D. Silva, “Towards Controlling Software Architecture Erosion
[5] A. L. Wolf and D. E. Perry, “Foundations for the Study of Software Through Runtime Conformance Monitoring,” Ph.D. dissertation, Uni-
Architecture,” ACM SIGSOFT Software Engineering Notes, vol. 17, versity of St Andrews, 2014.
no. 4, pp. 40–52, 1992.
[6] J. Knodel and D. Popescu, “A Comparison of Static Architecture
Compliance Checking Approaches,” in Proceedings of the Working
IEEE/IFIP Conference on Software Architecture (WICSA). IEEE, jan
2007.
Copyright © 2017 for this paper by its authors. 13