In this paper, we introduce various metrics for test coverage of clinical guidelines, modeled in the graphical language DiaFlux. Additionally, an intuitive visualization method supports the process of test creation and communicating the reached coverage levels to medical experts involved in the authoring of the guideline. The goal is to reach a sufficiently high test coverage to assure patient safety under all circumstances.
Testing is an important step in the development of any software
artifact, be it a program, a knowledge-based system, or a clinical
guideline. The two most prevalent testing strategies in Software
Engineering are black-box and white-box testing. The former approach is
unconcerned with the actual implementation and derives the test cases
solely from the underlying specification. The latter one, in contrast,
allows to create tests based on the implementation and to examine it
during execution of the tests. This introspection enables to capture
which basic elements of the tested artifact were executed - and thus
covered - by a test suite. Different metrics of Test Coverage were
developed to objectively measure and assess the thoroughness of such
testing efforts. In classic Software Engineering, metrics have been
defined to assess the coverage of, e.g., methods of a program,
statements of a method, taken decisions of control-flow, and so on [
Hence, the benefit of coverage metrics - and their proper visualization - is two-fold, increasing the effectiveness and efficiency of the test creation process: First, they help to avoid the creation of redundant tests. Second, they can be used to identify untested elements.
Both are also important aspects in the area of knowledge-based system in general and computerized clinical guidelines in particular. The creation of test cases for a clinical guideline will most likely involve both parties, the medical expert and the knowledge engineer. It is thus an expensive task, which should be completed efficiently. Though, the overall goal is to create a guideline, that assures patient safety under all circumstances, which can best be guaranteed by a thorough test suite. This is especially important in the area of automated guidelines, which are applied by closed-loop devices. They act autonomously on a patient to improve her state, without requiring constant supervision by a clinical user.
For the interpretation of coverage metrics a visualization is
helpful, especially to find untested elements. Test coverage of software
programs most usually is visualized by some kind of syntax
highlighting, by coloring, e.g., the executed statements. Though also
graphical representations were developed, e.g., [
In this paper, we introduce various metrics to determine the test coverage of clinical guidelines, modeled in the graphical language DiaFlux. Furthermore, we adapted the city metaphor by creating a CoverageCity for communicating the reached coverage levels to the involved medical experts in an accessible manner.
The rest of this paper is structured as follows: In the next section we give a short introduction into the graphical language DiaFlux for clinical guidelines. Section 3 presents coverage metrics for DiaFlux models. Following, in Section 4, we present the results of a case study. Finally, we conclude the paper with a summary and an outlook. 2
Clinical guidelines are an accepted means to improve patient outcome. Therefore, they offer a standardized treatment, based on evidence-based medicine. They are developed for several decades. In their beginnings, they were solely text-based documents that relied on the proper application by clinicians. The ongoing computerization and data availability, also in domains with high-frequency data as, e.g., Intensive Care Units (ICUs), allows for an automation of guideline application by medical devices.
Several formalisms for Computer-Interpretable Guidelines were
developed, every one with its own focus, like shareability between
institutions [
A downside of rule-based representations is their lower
comprehensibility compared to graphical ones. This especially holds true, as
medical experts are most usually involved in the creation of
guidelines. Therefore, we have developed a graphical guideline formalism
called DiaFlux [
The main application area of DiaFlux are mixed-initiative devices that continuously monitor, diagnose and treat a patient in the setting of an ICU. Such closed-loop systems interact with the clinical user during the process of care. Both, the clinician and the device, are able to initiate actions on the patient. Data is continuously available as a result of the monitoring task. It allows for repeated reasoning about the patient state, and to carry out appropriate actions to improve her condition, if necessary. 2.2
To specify a clinical guideline, two different types of knowledge
have to be effectively combined, namely declarative and
procedural knowledge [
In DiaFlux models, the declarative knowledge is represented by
a domain-specific ontology, which contains the definition of
findings and solutions. This application ontology is an extension of the
task ontology of diagnostic problem solving [
In the following, we informally describe the most important language elements:
Test node: Test nodes represent an action for the acquisition of data during runtime. This may trigger a question, the user has to answer, or data to be automatically obtained by sensors or from a database.
Solution node: Solution nodes are used to set the rating of a solution based on the given inputs. Established solutions generate messages for the clinical user and can, e.g., advice him to conduct some action.
Wait node: Upon reaching a wait node, the execution of the protocol is suspended until the given period of time has elapsed. Composed node: DiaFlux models can be hierarchically structured. Defined models can be reused as modules, represented by a composed node in the flowchart using it.
Abstraction node: Abstraction nodes offer the possibility to create abstractions from available data. These values can then be used for therapeutic actions by influencing the settings of the host device. 2.3
The execution engine for DiaFlux models is intended for, but not
limited to, closed-loop devices, that provide data from sensors or
manually entered by the clinical user and carry out therapeutic actions
on the patient, i.e., changing device settings in certain ranges. The
architecture of the DiaFlux guideline execution engine consists of
three components. First, a knowledge base, that contains the
application ontology and the flowcharts. Second, a blackboard, that stores
all findings about the current patient session. Third, a reasoning
engine, that executes the guideline and carries out its steps, depending
on the current state as given by the contents of the blackboard.
Therefore, the reasoning engine is notified about all findings, that enter the
blackboard. A more thorough introduction to the DiaFlux language
and its execution engine is given in [
The execution of the guideline is time-driven. The reasoning starts by acquiring data and by interpreting this data. The results are written to the blackboard. Then, the guideline is executed. This involves making decisions and possibly the generation of hints to the user and therapeutic actions by the device. Finally, the time of the next execution is scheduled, pausing the execution until that instance in time, waiting for the effects of the therapeutic interventions to take place. 3
Verification and validation of a clinical guideline are important steps
in its development. That way, patient safety shall be assured under
all circumstances. Verification usually consists of formal methods,
proofing, that a given guideline is free of internal inconsistencies and
incompleteness. Normally, these kinds of checks can be performed
without executing the guideline. An overview of verification
methods applied to clinical guidelines is given in [
In conventional software engineering (SE), test coverage (also
known as code coverage) is a well-established technique to measure
how well a piece of software is exercised by a test suite. Often, the
reached level of coverage is also used as an indicator for the quality
of the tested program, as tested elements are less likely to contain
errors than untested ones. Coverage criteria have been defined on
different levels of granularity, from the method-level down to single
statements, or even parts of them (so called condition coverage) [
In general, employing coverage metrics during the creation of a test suite may help improving it in terms of minimality and completeness. While a high coverage of the object under test is worthwhile, this should be accomplished with the possibly minimal set of test cases, as test creation is a difficult and costly task. This especially holds true for the test creation of clinical guidelines, as it may involve knowledge engineers as well as domain specialists like medical experts. Therefore, adopting the mentioned techniques for clinical guidelines and their graphical representations, offers the possibility to improve this process.
In the following, we introduce coverage metrics on different levels of granularity to assess the test coverage of a DiaFlux guideline. Such a guideline usually is modularized in self-contained modules, i.e. flowcharts. To represent this modularization also by the coverage metrics, we define most of them over the elements of individual flowcharts, hence the restriction of nodes and edges to those of a single one. This focusing alleviates the increase of coverage by additional tests, as deficiencies are easier to spot. Flowchart Coverage is defined as the ratio of the number of flowcharts that are executed by a test suite and the overall number of flowcharts in the guideline.
Let F be the set of DiaFlux models in guideline G, and FTex be the set of flowcharts executed by test suite T . Then, the Flowchart Coverage F CT of G is given by:
F CT (G) = jFTexj jF j This metric only gives a bird’s eye view of the testing situation. It can be used to guarantee at least some testing in all areas of the guideline during the starting phase of test creation. As it is a very coarsegrained measurement, a F CT value of 100% should be aimed at. Otherwise, major parts of the guideline remain untested. In SE, the equivalent metric is function coverage, which reports, if every function of a program has been tested. 3.2
Nodes represent the elementary steps of a guideline. A node being covered by a test suite, means, that its associated guideline step has been carried out at least once during the execution of the test suite.
Let Nf ; f 2 F be the set of nodes of the flowchart f , and Nfe;xT ; f 2 F be the set of nodes of the flowchart f , that are executed by test suite T . The according Node Coverage metric N CT of a flowchart f for a given test suite T can then be calculated as: N CT (f ) = jNfe;xT j ; f 2 F jNf j Similar to Flow Coverage, a N CT level of 100% should be reached for every flowchart in the guideline, as untested nodes can enact actions with unforeseen effects. This metric is equivalent to Statement Coverage in classic SE, which reports, if every statement of a tested function has been executed. 3.3
Edges are used to create the control flow of a flowchart, by defining paths of possible sequences of guideline steps. Every node can have several outgoing edges. Each of these edges can be guarded by a condition, to select the appropriate successor node, depending on the outcome of each guideline step. Therefore, the Edge Coverage metric reports, if all possible outcomes of the guideline steps - in terms of the equivalence classes that are defined by the edge guards - have been considered within the test suite.
Let Ef ; f 2 F be the set of edges of flowchart f , and Efe;xT ; f 2 F be the set of edges of the flowchart f , that are executed by test suite T . Then, ECT is defined as the number of activated edges of a flowchart f to their overall number, executing a given test suite T : ECT (f ) = jEfe;xT j ; f 2 F
jEf j As edges connect nodes, an Edge Coverage subsumes the Node Coverage of the according flowchart. This metric can be compared to Decision Coverage in SE, that keeps track, if each decision in a program under test (e.g., if- and switch-statements) has at least once been taken and once not. 3.4
An edge guard may not be an atomic condition, but consist of several sub-conditions, connected by boolean operators. For such nonatomic guards, Edge Coverage gives no detailed information about which of its sub-conditions were satisfied and which were not. This is of special interest, if the sub-conditions are joined by an OR-operator. In this case, every possible combination of atomic conditions, that can fulfill the overall condition, have to be tested.
A more detailed view about this issue is given by Condition
Coverage. It checks, if every atomic condition has once been satisfied
and once not. In classical SE, several different metrics for this issue
have been developed, e.g. Modified Condition / Decision Coverage
[
Let Pf ; f 2 F be the set of paths through flowchart f , and Pfe;xT ; f 2 F be the set of paths through flowchart f , that are executed by test suite T . Then, P CT is calculated as the number of paths taken through flowchart f , by the execution of test suite T , divided by the total number of paths:
P CT (f ) = jPfe;xT j ; f 2 F
jPf j
Even with a proper modularization given, not every modeled path may be enactable, due to implicit dependencies between the guideline steps. If certain combinations of decisions and actions on a single path exist, that can not occur, the targeted value of Path Coverage has to be decreased accordingly.
As a path consists of consecutive nodes and edges, Path Coverage satisfies Node Coverage as well as Edge Coverage.
Path Coverage, as defined above, is a rather aggregated measurement and thus gives little advice of how to improve coverage with further tests. Therefore, Path Coverage can also be restricted to the paths through a specific node n 2 N :
Let Pn be the set of paths containing n (8p 2 Pn : n 2 p), and Pne;xT be the set of paths containing n exercised by test suite T . Accordingly, the Path Coverage of node n is defined as: P CT (n) = jPne;xT j jPnj
A node with a low Path Coverage is only tested under a small fraction of the contexts in which it is contained. Again, further tests should be created for those scenarios, unless they expose dependencies that makes not every path feasible. 3.6
The metrics presented so far assess the test coverage with respect to structural properties, each considering some kind of modeled element, like nodes and edges. However, the actual input data, that directs the execution of the guideline, is not assessed by these metrics in any way.
Beside the mentioned control-flow-based metrics, a second
perspective on coverage in classic Software Engineering is given by
data-flow-based metrics. Those measure the coverage of
definitionuse (du) sequences of variables, i.e., a block of instructions in which
a variable is defined and subsequently used without a redefinition of
the variable, e.g., [
DiaFlux models do not contain variables as they are common in procedural programming languages, and the input data is not modified during guideline execution. Therefore, a definition-use analysis is not applicable. Equivalence Partitioning and Boundary Value Analysis can also not be used as they are. Explicit equivalence classes usually can not be stated for inputs. Even thresholds for less determined assessments (like “low”, “normal”, “high”) are often hard to specify for a medical expert. Those can furthermore vary between different types of patients, which, e.g., share an underlying disease. However, for each numerical input, a contiguous interval of possible values can typically be given, according to the human body’s physiological system and/or the preconditions of guideline applicability. To assure a proper coverage of allowed input data regardless of concepts like equivalence classes, we define the metric Input Coverage: Let the interval [mini; maxi] be the domain Di for numerical input i, and n 2 N; n > 0 be the number of equally-sized partitions of Di. The function cover(i; j) returns 1, iff the j th partition of Di contains at least one input value in test suite T, and 0 otherwise. Then, the Input Coverage of i is given by: n
P cover(i; j) ICn;T (i) = j=1 n
Clearly, the significance of Input Coverage depends on the actual value of n. It should be chosen to appropriately represent the sensitivity of the outcome of the guideline to changes in values of i. At later stages of test creation, the value can be increased stepwise to test more fine-granular in terms of i’s input values.
Similarly, the output of the guideline (which mainly consists of numerical values of the host device’s settings in predefined ranges) can be assessed by the analog defined metric Output Coverage OCn;T . 3.7
Commonly, there are two strategies to gather the data needed for calculating test coverage metrics. The first one is called instrumentation, which modifies the tested piece of software by including new code that collects the necessary information. The second strategy is tracing, which traces the executed elements by using some sort of debugging API (Application Programming Interface) of the execution environment. Clearly, both approaches have an effect on the execution time of the tests, as additional data has to be gathered. An advantage of tracing is, that it does not alter the executed artifact. Under certain circumstances this may also influence its behavior. Under this aspect, “tracing” seems preferable, if the necessary API is available. 3.8
The calculated metrics result in a numerical value representing the
test coverage of the exercised artifact. This may very well be
comprehensible for software and knowledge engineers, though for
nontechnical domain specialists, like medical experts, these sole
numbers may not be accessible enough. Furthermore, only a proper
composition of metrics yields a meaningful overall picture, as each metric
represents a different aspect of coverage. Thus, an intuitive
visualization as a means for communicating the reached coverage levels
to domain specialists seems preferable. One approach to this need
are so called “Polymetric Views” [
This section describes an implementation of coverage metrics for DiaFlux guidelines and a small case study. 4.1
The development environment for DiaFlux guidelines is integrated into the Semantic Wiki KnowWE. We created a plugin to calculate the test coverage of DiaFlux models, when executing a test suite. It employs the tracing approach to collect the information about exercised elements of the guideline. For each execution of the guideline, the chosen path according to the input data is recorded. After finishing the test suite, the metrics are calculated and can subsequently be visualized as CoverageCity, which can freely be scaled and rotated (cf. Figure 2). For creating the visualization, we used the WebGL4based JavaScript library SceneJS5. It is hence accessible with every (modern) web browser from within KnowWE, not requiring any proprietary software. 4.2
For an accessible visualization of the reached coverage levels, we use
the metaphor of a city. It has been introduced as graphical
representation of static code metrics (e.g. number of methods,. . . ) in the
context of reverse-engineering of software systems (“CodeCity”) [
We adapted the city metaphor for the visual representation of test coverage of DiaFlux guidelines. Districts stand for individual flowcharts. They are nested as given by their call hierarchy. Buildings correspond to the nodes of the district’s corresponding flowchart. Edges are not explicitly included but mapped to one of the buildings’ dimensions. In particular, we used the following assignment of metrics to visual properties of buildings:
Length: The number of outgoing edges of a node determines the length of the building.
Width: The width of a building relates to the number of outgoing edges that are covered by the test suite. 4 http://webgl.org 5 http://scenejs.org Height: The height of a building shows how often it was covered by the test suite.
Color: The color represents how well the paths, that go through a particular node, are covered by a test suite. In case a building is “red”, only few paths are executed. “Green” stands for a high Path Coverage. 4.3
The complexity that a node introduces into a guideline, mostly comes from its number of outgoing edges. In case the associated action has numerous possible outcomes (each represented by an outgoing edge), it is more likely to contain errors. Thus, one dimension of the base area of a building is influenced by the number of outgoing edges of the corresponding node. The increased size makes the building easier to spot. The other dimension of the base area grows with the number of covered outgoing edges. As a result, a non-quadratic building stands for a node, whose outgoing edges are not completely covered. The lower the aspect ratio, the more uncovered edges exist. This deficiency in test coverage can easily be observed.
The color of a building depicts its Path Coverage. A red one, represents a node, that is contained in much more paths than were covered. With increasing Path Coverage, the color becomes green. The height of a building corresponds to the number of times, it has been exercised by the test suite. Clearly, those two properties are not independent. On the one hand, a small building is more likely to be contained only in a small number of paths, and thus be red. On the other hand, a building that is tall and red, implies that the corresponding node is often tested under similar circumstances. This can give hints for creating new test cases, that are not contained so far. Tall, green buildings have been thoroughly exercised and do not require additional test cases.
The nesting level of the districts helps in estimating the necessary effort for testing a specific flowchart or node. Deeply nested module are probably harder to reach by a test case, as each module may only be called under certain preconditions.
The aggregated view of CoverageCity makes it easy to spot deficiencies in test coverage quickly. Though, a more detailed view is necessary to identify the specific nodes and their test deficiencies. Hence, each building be selected by the user. Then, the corresponding flowchart is shown below the CoverageCity, and the node and its coverage is highlighted visually. 4.4
Currently, we are involved in the implementation of a computerized
guideline for automated mechanical ventilation [
We selected a sample of ten generated test cases. The visualization of the acquired coverage levels is shown in Figure 2. Currently, we are in the process of evaluating our visualization with medical experts. Furthermore, we identify other meaningful assignments of metrics to visual properties of the buildings. 5
In this paper, we formally defined different coverage metrics to assess the thoroughness of testing efforts for clinical guidelines. They can be used to identify insufficiently tested elements, and to improve the process of test creation in terms of efficiency, as this may involve domain specialists as well as knowledge engineers. An intuitive visualization method helps in communicating the acquired coverage levels to domain specialists, for which numerically expressed metrics probably are less helpful than for knowledge engineers.
Additional metrics could be defined over more dynamic aspects of a guideline. First, the distribution of values could be tracked for each activated flowchart element. As there clearly are dependencies between the actual values and the possible ones - given by the context (i.e. path) of the element - proper preprocessing would be necessary. Ultimately, this could give insight, if parts of the guideline were only tested, e.g., for a certain patient type. Second, it would be helpful to define scenarios with respect to the occurrence of a certain sequence of input data or therapeutic actions over time and trace their coverage by the test suite. In terms of the CoverageCity-visualization, we will evaluate different mappings of the coverage metrics to the visual properties of the city, to create new perspectives on test coverage.
One shortcoming of white-box testing in general is, that it is
unable to detect errors by omission, i.e. some requirement may not have
been included in the implementation under test. An approach to find
this type of errors is Requirements-based Test Coverage [
University of Wu¨ rzburg is funded by the German Federal Ministry for Education and Research under the project “WiMVent” (Knowledge- and Model-based Ventilation), grant number 01IB10002E.