This poster presents the latest results from a very large eye tracking study (n=29) that explores how modelers read UML diagrams. We nd that various factors like layout quality, modeler experience, and diagram type lead to signi cant di erences in diagram reading strategies. We derive elements of a theory of diagram reading behavior from our ndings. This paper presents only late breaking results: all ndings presented, theories constructed, and conclusions drawn are of a preliminary nature. This paper does not present the amount and degree of evidence that would allow us to consider the contents as being scienti cally validated, yet.
Practical experience suggests that usage and understanding of UML diagrams
is greatly a ected by the quality of their layout: it is a lot harder to spot errors
and weaknesses in a cluttered diagram, and it is a lot easier to understand the|
quite literally|overall picture of a nicely laid out diagram. Previous research
has identi ed what constitutes good and bad practices of layout (see e.g. [
\We set up an experiment to study the impact of [layout] rules on
understandability [...] operationalized in terms of [...] faults and [...] time. We could
not identify a statistically signi cant relation between [them]."[
On the other hand, our own previous work provided substantial evidence to
this e ect [
Since there is such a stark contrast between our results and the previous literature, we deemed it necessary to add to the validity of our study by a replication of the results. The replication varies the previous studies in three aspects.
{ First, obviously, di erent participants were recruited for the present study.
They are of a comparable population, though, so this amounts to a minor variation point only. { Second, the study is designed and conducted by two graduate students (the second and third authors), so as to reduce impact and bias through the person of the experimenter of the previous studies (the rst author). Again, this was expected to be of secondary importance due to the previous study setup that had reduced personal interactions almost entirely. { Third, we use a di erent method (eye tracking) with objective, physiological measurements of cognitive load, thus addressing critique that subjective assessments of cognitive load are not su ciently reliable. We also used the measurements previously employed (score, duration, subjective assessment) to establish a point of reference.
Next we were curious whether or not we would be able to observe any
identi able behavior patterns, that amount to reading strategies. If such reading
strategies do exist, would there be di erences under the various treatments, e.g.,
would novices exhibit di erent reading strategies than experts, say? Ideally, this
should lead to elements of a theory of how UML models (and similar artifacts)
are processed by human modelers, and what we can do to support this process.
Since these are subtle questions, and since we have not used eye tracking as a
research method before, we have developed and re ned our study design and
experimental procedure in a pilot study, see [
By and large, the present study produced the same results as previous studies: the impact of layout quality on modeler performance could be replicated, though it showed more in the objective measures (score, time) than in subjective e ort assessment { in our earlier studies, we found a stronger e ect in cognitive load than in objective achievement. The di erences are not so big as to be inexplicable, though, which leads us to attribute it to the di erent experimental setting, and the di erent populations. As in earlier experiments, the e ect appears independent of diagram type, but is in uenced by the expertise level, and the size of diagrams.
Previously, we had only used subjective assessments of cognitive load which
have been criticized as less reliable than physiological indicators like heart rate
or skin conductivity. So we instrumented our stimuli accordingly and measured
the pupil dilation, xation duration, and blink rate associated to diagram
elements and easily localizable layout problems (e.g., line crossings, line bends).
We could clearly show that subjective assessments correlate tightly to
physiological measures, which is in line with existing literature on the subject [
We could clearly identify behavioral patterns in the subjects reading diagrams. In order to study these patterns, we recorded and analyzed the starting point and progress of participants' scan paths. We found three di erent behavioral patterns in them.
{ Branch Following From Anchor (BFFA): An anchor would be established either at the top left corner, the largest element, or the center of the diagram. Starting from there, the graph structure of the diagram would be followed in a systematic way. { Left/Right Top/Bottom (LRTB): starting at the top left corner, proceed downwards and to the right irrespective of the direction of arrows in the diagram, i.e., as if reading a text in the normal reading direction for western languages. { Random Walk (RW): Starting at any point, continue to any other point and so on, at no discernible pattern.
LRTB coincides with BFFA for diagrams of a diagram type with natural direction (e.g., Activity Diagrams), when the actual diagram layout agrees with the reading direction. In the literature, this has been described as \diagram ow", a notion that could be formalized as the dominant direction of directed elements in a diagram. This de nition does not cover other kinds of ow like circular or radial, but our study did not consider examples of these layout types.
Interestingly, all participants (i.e., both experts and novices) followed the BFFA or LRTB strategies for diagrams with good layout, irrespective of diagram type. For diagrams with bad layout, however, novices tended to use the RW strategy while experts continued to use the BFFA/LRTB strategies. 4
The stability of behavioral patterns we have observed were relatively stable over di erent individuals, and even more so for advanced modelers. This might be interpreted to show that while behavior exhibits individual variation in less skilled modelers, increasing experience results in a converge of behavior. If this is so, it is likely that one and the same cognitive process is e ective universally, in all modelers|but what can we know about this process?
In a recent study [
The literature on layout quality, however, frequently discusses two distinct layers: one concerned with low-level layout aws like line crossings, and one concerned with higher-level aspects like ow (this latter level is sometimes called diagram architecture). And indeed, as we have seen above, there are clearly discernible reading patterns that obviously are independent of diagram size, but do depend on the overall layout structure. So, there is some indication that diagram reading might involve at least two di erent cognitive processes, but it is not quite clear whether and how they are connected.
For diagram types with a \natural" ow like Activity Diagrams, it is fairly obvious to assume that following the reading direction (strategy BR) leads to good layouts. So we might be tempted to consider \layout ow" as a layout quality measure. But what does ow mean for diagrams with no natural directionality? Also, our sample did not contain diagrams with radial or circular layout where di erent ow might be observed. Finally, it might be that ow is the wrong notion to begin with, and we should rather consider ow as a special kind of symmetry, and explore this notion instead. This will require experiments on speci cally created layouts that highlight one or the other aspect, and to see whether these layout styles give rise to di erent behavior. 5
We conduct a study on the impact of layout quality, expertise level, and diagram type on the understanding of UML diagrams. Like in earlier studies, we nd that the diagram understanding outcome (measured in time, errors, and load) is a ected by layout quality, even though in the present experiment, we used eye tracking instead of the subjective assessments of cognitive load we had used before. We conclude that our initial study and our replication corroborate each other, underlining the validity of the evidence presented.
But it is not just the outcome that is a ected by layout quality, we also observe changes to the diagram reading behavior (measured as start location and subsequent evolution of the scan path). Even though the results presented here are of preliminary nature they clearly indicate that there are several distinct reading strategies. Surprisingly, we nd the same reading strategies for di erent diagram types: one would expect that diagram types with a natural directionality might be read in a di erent way than diagrams without a \built-in" direction. More research is needed to explore this question further, possibly the experimental setup of our study (in particular, the respective layouts) limit the insight of our study.
We have replaced the global and undi erentiated assessment of cognitive load
used in earlier experiments by three concurrent, objective measurements with
high temporal and spatial resolution: pupil dilation, xation duration, and blink
rate. This allows us to identify the cognitive load of individual diagram elements,
and we observe that shapes (i.e., diagram elements representing classes, use
cases, actions, and so on) provoke a similar cognitive load pro le than layout
aw such as line crossings and line bends. In other words: increasing the number
of diagram elements has the same e ect on cognitive processing than increasing
the number of layout aws. This explains in a very elegant way our previous
nding that diagram size seems to be more important than layout quality: the
number of diagram aws can (often) be reduced, the diagram size (measured in
the number of elements, [
We replicate earlier experiments on the impact of layout quality (and other factors) on UML diagram understanding by using eye tracking instead of subjective assessment. We observe characteristic behavior patterns and study several in uence factors. We observe a great uniformity in the reading strategies applied (at least for experienced modelers) to the degree we are almost certain to observe a universal cognitive process. If it is indeed universal, we might be able to nd a physiological substrate to the observed behavior. Obviously, our present study is not suited to explore this thread further, and we have to defer that to future research employing di erent machinery, in particular brain imaging.