The Impact of Hierarchy on Model Understandability
In this section we revisit results from prior experiments on the in uence of
hierarchy on model understandability, and analyze them from a cognitive perspective.
Sect. 2.1 summarizes literature reporting experimental results. Sect. 2.2 describes
cognitive foundations of working with hierarchical models.
2.1
Existing Empirical Research into Hierarchical Models
The concept of hierarchical structuring is not only applied to various domains,
but also known under several synonyms. In particular, we identi ed synonyms
hierarchy, hierarchical, modularity, decomposition, re nement, sub-model,
subprocess, fragment and module. Similarly, model understandability is referred to
as understandability or comprehensibility. To systematically identify existing
empirical investigations into the impact of hierarchy on understandability within
the domain of conceptual modeling, we conducted a systematic literature
review [
11
]. More speci cally, we derived the following key-word pattern for our
search: (synonym modularity) X (synonym understandability) X experiment X
model. Subsequently, we utilized the cross-product of all key-words for a full-text
search in the online portals of Springer1, Elsevier2, ACM3 and IEEE 4 to cover
the most important publishers in computer science, leading to 9,778 hits. We
did not use any restriction with respect to publication date, still we are aware
that online portals might provide only publications of a certain time period. In
the next step, we removed all publications that were not related, i.e., did not
consider the impact of hierarchy on model understandability or did not report
1 http://www.springerlink.com
2 http://www.sciencedirect.com
3 http://portal.acm.org
4 http://ieeexplore.ieee.org
empirical data. All in all, 10 relevant publications passed the manual check,
resulting in the list summarized in Table 1. Having collected the data, all papers
were systematically checked for the in uence of hierarchy. As Table 1 shows,
reported data ranges from negative in uence [
12
] over no in uence [12{14] to
mostly positive in uence [
15
]. These experiments have been conducted with a
wide spectrum of modeling languages. It is interesting to note though that
diverse e ects have been observed for a speci c notation such as statecharts or
ER-models. In general, most experiments are able to show an e ect of hierarchy
either in a positive or a negative direction. However, it remains unclear under
which circumstances positive or negative in uences can be expected. To approach
this issue, in the following, we will employ concepts from cognitive psychology
to provide a systematic view on which factors in uence understandability.
Work Findings
Moody [
15
] Positive in uence on accuracy, no in uence /
negDomain: ER-Models ative in uence on time
Reijers et al. [
16, 17
] Positive in uence on understandability for one out
Domain: Business Process Models of two models
Cruz-Lemus et al. [
9, 18
] Series of experiments, positive in uence on
underDomain: UML Statecharts standability in last experiment
Cruz-Lemus et al. [
13
] Hierarchy depth of statecharts has no in uence
Domain: UML Statecharts
Shoval et al. [
14
] Hierarchy has no in uence
Domain: ER-Models
Cruz-Lemus et al. [
8
] Positive in uence on understandability for rst
Domain: UML Statecharts experiment, negative in uence in replication
Cruz-Lemus et al. [
12, 19
] Hierarchy depth has a negative in uence
Domain: UML Statecharts
As discussed in Sect. 2.1, the impact of hierarchy on understandability can range
from negative over neutral to positive. To provide explanations for these diverse
ndings, we turn to insights from cognitive psychology. In experiments, the
understandability of a conceptual model is usually estimated by the di culty of
answering questions about the model. From the viewpoint of cognitive
psychology, answering a question refers to a problem solving task. Thereby, three di erent
problem-solving \programs" or \processes" are known: search, recognition and
inference [
20
]. Search and recognition allow for the identi cation of information
of low complexity, i.e., locating an object or the recognition of patterns. Most
conceptual models, however, go well beyond complexity that can be handled
by search and recognition. Here, the human brain as a \truly generic problem
solver" [
21
] comes into play. Any task that can not be solved by search or
recognition, has to be solved by deliberate thinking, i.e., inference, making inference the
most important cognitive process for understanding conceptual models. Thereby,
it is widely acknowledged that the human mind is limited by the capacity of its
4
working memory, usually quanti ed to as 7 2 slots [
22
]. As soon as a mental
task, e.g., answering a question about a model, overstrains this capacity, errors
are likely to occur [
23
]. Consequently, mental tasks should always be designed
such that they can be processed within this limit; the amount of working memory
a certain task thereby utilizes is referred to as mental e ort [
24
].
In the context of this work and similar to [
25
], we take the view that the
impact of modularization on understandability, i.e., the in uence on inference,
ranges from negative over neutral to positive. Seen from the viewpoint of
cognitive psychology, we can identify two opposing forces in uencing the
understandability of a hierarchically structured model. Positively, hierarchical structuring
can help to reduce the mental e ort through abstraction by reducing the
number of model elements to be considered at the same time [
15
]. Negatively, the
introduction of sub-models may force the reader to switch her attention between
the sub-models, leading to the so-called split-attention e ect [
26
]. Subsequently,
we will discuss how these two forces presumably in uence understandability.
Abstraction. Through the introduction of hierarchy it is possible to group a part
of a model into a sub-model. When referring to such a sub-model, its content
is hidden by providing an abstract description, such as a complex activity in a
business process model or a composite state in an UML statechart. The
concept of abstraction is far from new and known since the 1970s as \information
hiding" [
1
]. In the context of our work, it is of interest in how far abstraction
in uences model understandability. From a theoretical point of view, abstraction
should show a positive in uence, as abstraction reduces the amount of elements
that have to be considered simultaneously, i.e., abstraction can hide irrelevant
information, cf. [
15
]. However, if positive e ects depend on whether information
can be hidden, the way how hierarchy is displayed apparently plays an
important role. Here, we assume, similar to [
15, 17
], that each sub-model is presented
separately. In other words, each sub-model is displayed in a separate window if
viewed on a computer, or printed on a single sheet of paper. The reader may
arrange the sub-models according to her preferences and may close a window or
put away a paper to hide information. To illustrate the impact of abstraction,
consider the BPMN model shown in Fig. 1. Assume the reader wants to
determine whether the model allows for the execution of sequence A, B, C. Through
the abstraction introduced by sub-processes A and C, the reader can answer this
question by looking at the top-level process only (i.e., activities A, B and C);
the model allows to hide the content of sub-processes A and C for answering this
speci c question, hence reducing the number of elements to be considered.
Split-Attention E ect. So far we have illustrated that abstraction through
hierarchical structuring can help to reduce mental e ort. However, the introduction
of sub-models also has its downsides. When extracting information from the
model, the reader has to take into account several sub-models, thereby
switching attention between sub-models. The resulting split-attention e ect [
26
] then
leads to increased mental e ort, nullifying bene cial e ects from abstraction.
In fact, too many sub-models impede understandability, as pointed out in [
4
].
Again, as for abstraction, we assume that sub-models are viewed separately. To
illustrate this, consider the BPMN model shown in Fig. 1. To assess whether
activity J can be executed after activity E, the reader has to switch between
the top-process as well as sub-processes A and C, causing her attention to split
between these models, thus increasing mental e ort.
A
B
C
D
E
F
H
J
I
While the example is certainly arti cial and small, it illustrates that it is
not always obvious in how far hierarchical structuring impacts a model's
understandability.5
3
Assessing the Impact of Hierarchy
Up to now we discussed how the cognitive process of inferencing is in uenced by
di erent degrees of hierarchical structuring. In Sect. 3.1, we de ne a theoretical
framework that draws on cognitive psychology to explain and integrate these
observations. We also discuss the measurement of the impact of hierarchy on
understanding in Sect. 3.2 along with its sensitivity to model size in Sect. 3.3
and experience in Sect. 3.4. Furthermore, we discuss the implications of this
framework in Sect. 3.5 and potential limitations in Sect. 3.6.
Model
Subject
about question
answers
has
hierarchy
influences
yields
answer
influences
estimates
model
understandability
5 At this point we would like to remark that we do not take into account class diagrams
hierarchy metrics, e.g. [
27
], since such hierarchies do not provide abstraction in the
sense we de ne it. Hence, they fall outside our framework.
6
thereby is how the hierarchy of a model in uences understandability. In order
to operationalize and measure model understandability, a common approach is
to use the performance of answering questions about a model, e.g., accuracy or
time, to estimate model understandability [
9, 17, 18
]. In this sense, a subject is
asked to answer questions about a model; whether the model is hierarchically
structured or not serves as treatment.
When taking into account the interplay of abstraction and split-attention
effect, as discussed in Sect. 2.2, it becomes apparent that the impact of hierarchy
on the performance of answering a question might not be uniform. Rather, each
individual question may bene t from or be impaired by hierarchy. As the
estimate of understandability is the average answering performance, it is essential
to understand how a single question is in uenced by hierarchy. To approach this
in uence, we propose a framework that is centered around the concept of mental
e ort, i.e., the load imposed on the working memory [
24
], as shown in Fig. 3. In
contrast to most existing works, where hierarchy is considered as a dichotomous
variable, i.e., hierarchy is present or not, we propose to view the impact of
hierarchy as the result of two opposing forces. In particular, every question induces
a certain mental e ort on the reader caused by the question's complexity, also
referred to as intrinsic cognitive load [
23
]. This value depends on model-speci c
factors, e.g., model size, question type or layout, and person-speci c factors, e.g.,
experience, but is independent of the model's hierarchical structure. If hierarchy
is present, the resulting mental e ort is decreased by abstraction, but increased
by the split-attention e ect. Based on the resulting mental e ort, a certain
answering performance, e.g., accuracy or time, can be expected. In the following,
we discuss the implications of this framework. In particular, we discuss how to
measure the impact of hierarchy, then we use our framework to explain why
model size is important and why experience a ects reliable measurements.
question complexity
induces
abstraction
lowers
mental effort
determines
performance
enables
increases
hierarchy causes
split-attention effect
As indicated [
9, 8, 15, 17, 18
] it is unclear whether and under which circumstances
hierarchy is bene cial. As argued in Sect. 2.2, hierarchical structuring can a ect
answering performance positively by abstraction and negatively by the
splitattention e ect. To make this trade-o measurable for a single question, we
provide an operationalization in the following. We propose to estimate the gains of
abstraction by counting the number of model elements that can be \hidden" for
answering a speci c question. Contrariwise, the loss through the split-attention
e ect can be estimated by the number of context switches, i.e., switches
between sub-models, that are required to answer a speci c question. To illustrate
the suggested operationalization, consider the UML statechart in Fig. 4. When
answering the question whether sequence A, B is possible, the reader presumably
bene ts from the abstraction of state C, i.e., states D, E and F are hidden|
leading to a gain of three (hidden model elements). On the contrary, when
answering the question, whether the sequence A, D, E, F is possible, the reader
does not bene t from abstraction, but has to switch between the top-level state
and composite state C. In terms of our operationalisation, no gains are to be
expected, since no model element is hidden. However, two context switches when
following sequence A, D, E, F, namely from the top-level state to C and back,
are required. Overall, it can be expected hierarchy compromises this question.
A
X
Y
B
C
W Z
D
F
E
Regarding the use of this operationalization we have two primary purposes in
mind. First, it shall help experimenters to design experiments that are not biased
toward/against hierarchy by selecting appropriate questions. Second, on the long
run, the operationalization could help to estimate the impact of hierarchy on a
conceptual model. Please note that these applications are to be viewed under
some limitations as discussed in Sect. 3.6.
3.3
Model Size
Our framework de nes two major forces that in uence the impact of
hierarchy on understandability: abstraction (positively) and the split-attention e ect
(negatively). In order that hierarchy is able to provide bene ts, the model must
be large enough to bene t from abstraction. Empirical evidence for this theory
can be found in [
9
]. The authors conducted a series of experiments to assess
the understandability of UML statecharts with composite states. For the rst
four experiments no signi cant di erences between attened models and
hierarchical ones could be found. Finally, the last experiment showed signi cantly
better results for the hierarchical model|the authors identi ed increased
complexity, i.e., model size, as one of the main factors for this result. While it seems
very likely that there is a certain complexity threshold that must be exceeded,
so that desired e ects can be observed, it is not yet clear where exactly this
threshold lies. To illustrate how di cult it is to de ne this threshold, we would
like to provide an example from the domain of business process modeling, where
estimations range from 5{7 model elements [
5
] over 5{15 elements [
6
] to 50
elements [
7
]. In order to investigate whether such a threshold indeed exists and
how it can be computed, we envision a series of controlled experiments. Therein,
we will systematically combine di erent model sizes with degrees of abstraction
and measure the impact on the subject's answering performance.
8
3.4
Besides the size of the model, the reader's experience is an important
subjectrelated factor that should be taken into account [
28
]. To systematically answer
why this is the case, we would like to refer to Cognitive Load Theory [
23
]. As
introduced, it is known that the human working memory has a certain capacity,
if it is overstrained by some mental task, errors are likely. As learning causes
additional load on the working memory, novices are more likely to make mistakes,
as their working memory is more likely to be overloaded by the complexity of
the problem solving task in combination with learning. Similarly, less capacity is
free for carrying out the problem solving task, i.e, answering the question, hence
lower performance with respect to time is to be expected. Hence, experimental
settings should ensure that most mental e ort is used for problem solving instead
of learning. In other words, subjects are not required to be experts, but must
be familiar with hierarchical structures. Otherwise, it is very likely that results
are in uenced by the e ort needed for learning. To strengthen this case, we
would like to refer to [
8
], where the authors investigated composite states in
UML statecharts. The rst experiment showed signi cant bene ts for composite
states, i.e., hierarchy, whereas the replication showed signi cant disadvantages
for composite states. The authors state that the \skill of the subjects using
UML for modeling, especially UML statechart diagrams, was much lower in this
replication", indicating that experience plays an important role.
3.5
Discussion
The implications of our work are threefold. First, hierarchy presumably does not
impact answering performance uniformly. Hence, when estimating model
understandability, results depend on which questions are asked. For instance, when
only questions are asked that do not bene t from abstraction, but su er from
the split-attention e ect, a bias adversely a ecting hierarchy can be expected.
None of the experiments presented in Sect. 2.1 describes a procedure for de ning
questions, hence inconclusive results may be attributed to unbalanced questions.
Second, for positive e ects of hierarchy to appear, presumably a certain model
size is required [
9
]. Third, a certain level of expertise is required that the impact
of hierarchy instead of learning is measured, as to be observed in [
8
].
3.6
Limitations
While the proposed framework is based on established concepts from cognitive
psychology and our ndings coincide with existing empirical research, there are
some limitations. First, our proposed framework is currently based on theory
only, an empirical evaluation is yet missing. To counteract this problem, we are
currently planning a thorough empirical validation, cf. Sect. 4. In this vein, also
the operationalization of abstraction and split-attention e ect needs to be
investigated. For instance, we do not know yet whether a linear increase in context
switches also results in a linearly decreased understandability, or the correlation
Assessing the Impact of Hierarchy on Model Understandability
9
can be described by, e.g., a quadratic or logarithmic behavior. Second, our
proposal focuses on the e ects on a single question, i.e., we can not yet assess the
impact on the understandability of the entire model. Still, we think that the
proposed framework is a rst step towards assessing the impact on model
understandability, as it is assumed that the overall understandability can be computed
by averaging the understandability of all possible individual questions [
29
].
4