Diagram-based visual languages are an established means for describing and developing software systems, but have an important drawback: users need to place diagram elements properly for a diagram to be properly readable, which can take a significant amount of time (Klauske and Dziobek 2010). Automatic layout algorithms aim to automate that task to improve user productivity. These algorithms enable new ways for working with visual languages, but present new challenges as well. In this paper, I suggest to think about automatic layout algorithms in terms of challenges and opportunities. Challenges are problems that need to be solved for layout algorithms to yield good results. Opportunities are problems inherent in visual languages or new user interaction scenarios that can be tackled with automatic layout. To illustrate these terms, I summarize previous research on one example of each.
Visual languages are popular among developers, both for describing existing software systems (UML being an example) and to develop new ones (languages such as LabVIEW,1 a system design and development platform). Visual languages have a reputation for being easy to understand and use, a natural way to communicate. After all, we quickly start scribbling diagrams on whiteboards or pieces of paper when try1 http://www.ni.com/labview ing to understand things or to explain them to others. If this serves us well, why then should computer-based visual languages be any different? For the purposes of this paper, we shall look at two approaches to answer that question.
First, there is the sheer size of diagrams. In contrast to diagrams scribbled on a sheet of paper, diagrams used to develop software systems can quickly grow so large that they cannot be displayed on a single screen without losing legibility. This forces developers to only look at small excerpts of the diagram at a time, which causes the mental overview to suffer and thereby decreases the diagram’s usability. Often enough, the problem is not just the number of diagram elements. Most visual languages contain text labels to identify elements and to specify behavior. Depending on the language, these labels can grow very large and force the diagram to grow very large as well. To make matters worse, long text labels do not simply enlarge a diagram; they do so by making it wider, causing the aspect ratio to grow less and less favorable for displaying the diagram on today’s common computer screens.
Second, a diagram is not easy to understand just because
it is visual in nature
Reducing that hit on developer productivity is what
automatic layout algorithms aim to do. Instead of having to place
all diagram elements manually, that task is delegated to the
computer. While this solves the problem of too much time
spent moving diagram elements around, it gives rise to new
problems when it comes to secondary notation. Consider a
diagram with two closely spaced elements. A human viewer
will immediately assume the two to be related in some way
and, when asked to modify the diagram, will probably try
to keep this visual clue intact. What a given automatic
layout algorithm will do in this situation depends strongly on
the particular algorithm. Misue et al.
Through the fast computation of layouts, automatic layout algorithms can serve as enablers for ways to improve the usability of visual languages. Let us come back to the problem of large text labels. Depending on what the user is trying to do at a given moment, the full content of one label may be important while not all the details of another label are of interest. Yet another label may not need to be shown at all, allowing the diagram to become smaller. However, what is and what is not interesting to a user changes dynamically as they concentrate on different parts of a diagram. This dynamic makes it impossible to exploit size changes with manual placement, but automatic layout is fast enough to quickly rearrange diagram elements to fit more of them on the screen. 1.1
In this paper, we take a step back from the details of how
automatic layout algorithms work and think about them
instead in terms of what keeps them from being used and what
new usage scenarios they enable. In this spirit, the main
contribution is the motivation and introduction of the notion
of challenges and opportunities of automatic layout
algorithms. This helps us bring together two formerly separate
lines of research we reported on previously: comment
attachment
The ultimate goal of our research is to improve the
usability of visual languages. It is thus worth recognizing that
other researchers have also found that there is, in fact, room
for improvement. This mainly affects two areas: the tooling,
and the visual languages themselves. While editing
environments for textual languages have matured to provide very
advanced and effective editing features, editors for visual
languages still leave something to be desired
It has been recognized that comments can relate to
diagram elements differently
The term label management and basic label shortening
techniques were introduced by Fuhrmann as part of his
dissertation
Label management aims to do something about the
problem of too little screen space and can be used to reduce the
amount of unnecessary information visible at a given time.
There are other techniques that focus on similar problems.
Fisheye views
We start by properly introducing what we mean by opportunities and challenges. This is followed by a discussion of two examples of opportunities and challenges, along with a summary of results obtained so far in these areas. We conclude with a discussion of the presented material.
The problem of placing diagram elements and routing edges
is a very complex one. Consider the three main phases of
the layered layout approach
A challenge is a problem inherent in a layout approach that needs to be solved in order for the approach to work or to improve the layout results it produces.
Research on layout algorithms usually focuses on efficient methods to tackle the complex and hard algorithmic problems, which of course are big challenges for algorithm developers. In the process, seemingly smaller and perhaps less theoretically interesting problems often seem to be overlooked and regarded as trivial implementation details. It is the experience at our research group, however, that if left unsolved these problems can be the reason why users fail to use automatic layout more than once. The experience that the layout algorithm has produced an unfortunate and obvious placement problem which is easily spotted and which “I could have easily done better” can lead users to never hit a layout button again. What’s more, many small and simple problems turn out to be not as small and simple once properly investigated. If layout algorithm developers want their algorithms to be widely adopted, both the obviously hard and the seemingly smaller challenges have to be addressed.
Problems are not to be found only in layout algorithms, but also in the design of visual languages. One example are long labels: if a visual language tends to make developers use long labels, the diagrams they produce will often grow too wide to be properly displayed on standard computer screens. This leads us to the following definition: An opportunity is a usability problem in the design of a visual language which can be mitigated through the use of automatic layout, or a new user interaction scenario enabled by automatic layout.
In the case of long labels, automatic layout algorithms can be used to shorten them and then modify the layout to take advantage of the saved space. Of course, taking advantage of such opportunities does pose additional problems to be solved on the layout algorithm side. In contrast to challenges, however, these problems are not inherent to the layout algorithms, but arise from problems with the languages that are to be laid out.
Starting with the next section, we will look at a challenge and an opportunity and summarize results obtained so far. Similar to textual languages, comments in visual languages help explain what is expressed through a diagram (see Figure 1). In textual languages it is usually very apparent what a comment refers to. Languages such as Java even make rules part of the language definition. There are usually no such rules in visual languages. If the comment is not explicitly connected to the diagram element it refers to (which not all languages support, and not all developers use), their relation is defined through more implicit and ill-defined means, such as how the comment is placed relative to other diagram elements, or the comment’s text.
If an automatic layout algorithm encounters a comment
with no explicit connection to the diagram element it relates
to, it may end up placing the two far apart. After all, it has no
idea that they should be placed in close proximity. We thus
need to make these relations available to layout algorithms.
This requires developing methods that recognize implicit
clues in order to turn them into explicit attachments between
comments and diagram elements that layout algorithms can
process. Computing these attachments is called the comment
attachment problem
Note that we would classify this as a challenge. Relating comments and diagram elements through placement in manual layouts works just fine, so this is not a problem of the visual language itself. The problem much rather lies in the layout algorithms that do not recognize these relations and may consequently fail to preserve them.
It is obvious that all comment attachment methods are
limited to be heuristics: there simply are no universal rules
people follow when writing and placing comments. Still, we
wanted to find out whether a number of obvious and
easyto-implement heuristics might be enough to infer the
relations between comments and diagram elements. In the past,
we have evaluated the performance of several such
heuristics
Font Size Text documents usually start with a heading set in a font size larger than the rest of the text. Similarly, if a diagram contains a comment whose font size is larger than that of all other comments, we might hypothesize that it contains the diagram’s title. Such a comment of course does not relate to any specific diagram element and should thus be left alone. This heuristic works well in that if it flags a comment as containing the diagram’s title, this is correct in 98% of all cases.
Text Prefix We found that comments that describe the diagram as a whole as opposed to a single element often start with a similar prefix, such as “This model. . . ”, or “Authors: . . . ”. Such comments do not relate to any single diagram element. How successful this heuristic is depends on how developers use a given visual language. It works well for the surveyed example diagrams in that if the heuristics flags a comment it is always correct, but it may fail for other languages.
Text References It seems obvious that if a comment actually mentions a diagram element by name, the two will probably be related. In practice, though, the comment may mention other elements as well, or the named element may just serve to distinguish it from the element the comment actually relates to.
Distance The distance between a comment and other diagram elements is probably the most obvious heuristic. However, the distance between a comment and the element it relates to is rather fuzzy in practice. Worse, often that element is not even the one closest to the comment. There is also the question of how far a comment and a diagram element may be apart for them to still be considered to be related.
Alignment Graphic designers use alignment as one of the main tools to establish relations between graphical elements. To our surprise, we found that developers working with visual languages care a lot less about alignment, thus rendering this heuristic almost useless.
In our evaluations, the best combination of heuristics was able to make correct decisions in about 90% of all cases. The heuristics that fared best were those based on established conventions for the given visual language, such as the font size and text prefix heuristic. Heuristics that rely more on developers having a grasp on graphic design are more problematic or even useless.
These results have yet to be verified with other visual languages. Still, they already allow for two observations. First, a good attachment result largely depends on how the heuristics are configured. The distance heuristic, for example, is very sensitive to the maximum distance that comments and diagram elements may be apart to still be considered related. Finding good settings can be challenging, especially since every team of developers can have different coding conventions. The second observation directly follows from this difficulty: inferring relations between comments and other diagram elements will always be prone to some margin of error. It therefore seems sensible for language designers to make explicit relations part of the language’s design, and to make it as easy as possible to define them through the tooling. 4.
Few visual languages can get away without incorporating
at least a bit of text. Languages based on state machines
usually use text labels to specify the conditions upon which
a transition can be taken as well as the actions executed
when it is. SCCharts
Label management aims to improve this situation by changing the content of labels and computing a new layout that takes advantage of the label size changes. The actual motivation to do so can vary:
Increase the zoom level. If the diagram should fit on the screen, a smaller diagram results in a larger zoom level, thereby increasing legibility. If the zoom level is fixed, smaller labels allow for more diagram elements to fit on the screen, thereby increasing the user’s overview. Reduce visual clutter. Whether the information a label contributes is of interest to the user or not depends on the task they are working on and what they are currently focusing on. Reducing the detail of or even removing unnecessary labels reduces visual clutter and allows more interesting elements to fit on the screen,
As mentioned in Section 2 this is an example of an opportunity. The problem to be solved—labels quickly becoming too large and increasing the size of diagrams—is inherent in the visual language. Automatic layout is used as an enabler for label management. After all, simply changing the size of labels does not accomplish much if we do not change the diagram’s size as well.
When reducing the size of labels two questions need to be answered: how do we shorten labels, and when do we do it?
Let us begin with the how. Fuhrmann
(hard) syntactical
(soft) semantical
Example (not SignalA) xor (...
SignalA, SignalB / SignalC (not SignalA) xor (not SignalB) / SignalC(cou nter) (not SignalA) xor (not SignalB) / SignalC( counter) (not SignalA) xor (not SignalB) / SignalC(counter) ing in what is required to get an idea of the original text. The three wrapping strategies are based in the observation that the width of labels is often more problematic than their height. Syntactical hard and soft wrapping work just like they do in text editors. Semantical wrapping makes sense in cases where labels follow a strict syntax: it restricts the points where line breaks can be inserted in an attempt to visually reflect the text structure.
We applied some of these heuristics to a set of 93 SCCharts containing a total of 6230 transition labels and measured the zoom level required to fit a whole diagram on the screen, with and without shortened labels. The results, shown in Figure 3, indicate that significant zoom level increases are possible. It does not come as a surprise that syntactical abbreviation is most effective in that regard: it always achieves the target length and does not increase a label’s height. Other strategies may be better, though, in terms of usability.
Let us now turn to the when of shortening labels, which will also answer the question of how to determine the tar[-] ThirdDude thirdDudeLogic @ OneDudeLogic [+] [-] SecondDude secondDudeLogic @ OneDudeLogic [+] readyState 1: cur entTime entryTime >= 300000 / indicatorReady = 0
apocalypseState entry / indicatorApocalypse = 255 exit / indicatorApocalypse = 0 (b)
With label management.
firstDude_request_ready == false | secondDude_request_ready == false | thirdDude_request_ready == false 2: firstDude_request_ready == true && secondDude_request_ready == true && thirdDude_request_ready == true firstDude_request_apocalypse == false && secondDude_request_apocalypse == false && thirdDude_request_apocalypse == false 1: firstDude_request_apocalypse == true | secondDude_request_apocalypse == true | thirdDude_request_apocalypse == true notQuiteBoringYetWeAreWorkingOnIt itsBoringThankYouVeryMuch
impendingDoomState entry / indicatorImpendingDoom = 255 exit / indicatorImpendingDoom = 0
zoom factor we achieved for our
with different label management
without label management.
get length. The most obvious way to decide which labels to
shorten is to use a preprocessing approach before the
automatic layout algorithm is triggered. The preprocessing can
make decisions based on information such as what parts of a
diagram are active while a simulation or debugging session
is running, or which diagram elements are selected. It is here
that the preprocessing approach can easily be integrated with
focus and context concepts
The preprocessing approach has the potential drawback that labels are shortened regardless of whether this actually has any impact on the diagram’s size. This is because the necessary information are not available until automatic layout is triggered. This is why we developed the feedback loop approach. Here, a preprocessing still configures how each label can be shortened, but the decision of whether or not it will be shortened is then left to the layout algorithm.
The two approaches also differ in how the target length for labels is determined. The preprocessing approach needs to set a target length more or less arbitrarily, perhaps based on user input. The feedback loop approach can allow labels to be as long as possible before having too much of an impact on their diagram’s size.
We have introduced the notions of opportunities and challenges in the context of automatic layout algorithms. To provide an example for each, we have summarized our findings regarding two lines of research: comment attachment (a challenge) and label management (an opportunity). The former served to illustrate how automatic layout algorithms are still posing challenges for algorithm designers when it comes to secondary notation. The latter served to illustrate, however, that automatic layout algorithms can also open up opportunities for new ways to help users accomplish their tasks, for example by auto-generating synthesized views that dynamically adapt to changing requirements.
We believe that algorithm designers often neglect the (seemingly) small challenges. In practice, however, we found that it is these details that keep users from embracing automatic layout. We also believe that we are just starting to realize the opportunities enabled by automatic layout algorithms.
There is much potential for future work. Regarding comment attachment, the methods need to be tested with more visual languages. Also, more complex heuristics could improve the quality of computed attachments. Regarding label management, a study will be required to show how label management impacts usability.