Understanding and Predicting the Affordances of Visual Logics

Abstract

We compare the affordances of two visual logics, one from the Euler family of notations, spider diagrams, and one which takes a significantly different approach to representing logical concepts, existential graphs. We identify strengths and weaknesses of each notation and present these features as being related to the idea that each notation is, to a greater or lesser degree, biased towards objects or predicates, and that such biases make a notation more or less effective in a given context. We then introduce a framework for understanding and predicting those affordances, which can help guide us towards better use of existing graphical notations and the design of more effective new notations. The framework links research in semiotics and linguistics with insights provided by the HCI and diagrams communities.

-

A fundamental premise of the diagrams community is that graphical notations have, by some set of metrics which is not always made entirely clear, certain advantages over symbolic notations. These advantages relate to intuitive understanding and to the ability for new information to arise spontaneously within diagrams. Gurr [ 10 ] wrote that the effectiveness of a graphical notation arises from its being “well matched to meaning”, which is to say that the syntax of the notation is naturally connected to its semantics. Hammer and Shin [ 11 ] showed that Euler diagrams do possess these advantages, and that while the changes made to Euler’s notation by Venn and Peirce remove ambiguity and increase formal expressiveness, they also reduce its visual clarity.

If these advantages exist, and can be categorised and measured, the potential exists to design more effective graphical notations and to make better use of existing ones. Shin [ 23 ] undertook the latter task in her reevaluation of Peirce’s system of existential graphs, in which she argued that if the diagrammatic properties of existential graphs were better understood and exploited in the design of reading procedures and inference rules, then they would be considered more useful as a tool for reasoning. Indeed, and in contrast to Euler diagrams, existential graphs have often been considered a cumbersome and non-intuitive system. In the same work, Shin shows, however, that Peirce consciously designed his system 3rd International Workshop on Euler Diagrams, July 2, 2012, Canterbury, UK. Copyright c 2012 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. to take advantage of distinctively diagrammatic properties, but that his insights were largely ignored in the way existential graphs were subsequently understood.

In this paper we will compare the affordances of two visual logics, spider diagrams [ 13 ] and existential graphs, analysing some of the strengths and weaknesses of each system. In this context, we use the term affordance to refer to the possible meanings of a piece of diagrammatic syntax, as perceived by an actor. The starting point for our comparison is the observation, made by Blackwell and Green [ 2 ], that “every notation highlights some kinds of information, at the cost of obscuring other kinds.” We focus on the affordances arising from the spatial conditions of diagrams from each system with the same meanings. Thus, we consider the static properties of the notations and how those properties support comprehension, rather than any dynamic properties exhibited when either notation is used as a reasoning system. This work is a precursor to a planned empirical study in which we will test our findings. Our goal is not to show that one notation is superior to the other. In fact, existential graphs are considerably more expressive than spider diagrams. To enable the comparison, we will consider the fragment of existential graphs with monadic predicates only, which is equivalent to the spider diagram system. We choose the two systems for the comparison because we take them to be representative of two distinct families of visual logics: those based on Euler diagrams, such as spider diagrams and constraint diagrams [ 24 ], and logical graphs, such as existential graphs and conceptual graphs [ 5 ]. Spider diagrams and existential graphs are concerned with the same domain and have common features. For instance, both represent existential quantification directly. Neither system has an explicit way of representing universal quantification but both can do so implicitly. However, the two systems take fundamentally different approaches to representing information.

In sections 2 and 3 we examine the notational strengths and weaknesses of spider diagrams and existential graphs. We do so informally, introducing only so much of each notation as is necessary to make our argument. In section 4 we introduce Coppin’s framework for visual affordances and show that it can be used to explain and predict the affordances described in the previous sections. The framework exposes general principles which we believe can be used to design effective visual notations, formal or otherwise. We show that the framework synthesises understandings gained from the fields of semiotics, neurolinguistics and diagrammatic reasoning. In section 5 we discuss the predictive power of the framework and the ways in which the principles of the framework may enable us to make more effective use of existing systems, such as spider diagrams and existential graphs, by understanding and exploiting their strengths. 2

Spider diagrams

Spider diagrams (SD) were introduced by Howse et al. in 2001 [ 13 ]. They are a sound and complete visual logic, equivalent in expressiveness to monadic firstorder logic with equality [ 25 ]. Figure 1 shows a spider diagram, consisting of labelled curves, spiders and shading. Curves represent sets and their placement makes assertions about relations between sets: figure 1 tells us that the sets Bird , Plane and Sman are disjoint. Spiders are trees placed in the diagram, where the nodes are called feet and the edges are called legs. Each spider represents a single element that exists in one of the regions in which its feet is placed. The diagram in figure 1 includes a single spider, telling us that there is something which is either a bird, a plane, or Superman. Shading is used to represent the emptiness of regions. So, the shading in figure 1, considered alongside the information provided by the spider, tells us that Sman contains either one element or no elements.

The spider diagrams in figures 1 and 2 are unitary diagrams. SD allows us to use conjunction and disjunction to join together unitary diagrams to form compound diagrams. This is done by placing the usual symbols from symbolic logic in part of the diagram: see Howse et al. [ 13 ] for details.

As well as statements about sets, we can construct the meaning of a spider diagram as a series of logical assertions. From this point of view the diagram in figure 1 states, among other things, that ∃x (Bird (x) ∨ Plane(x) ∨ Sman(x)), and ∀x (Bird (x) ⇒ ¬Plane(x)).

SD extends the Euler diagram notation which, as noted in the previous section has been identified as being intuitive or fit-for-purpose as the basis of diagrammatic reasoning by several authors (see for, instance, chapter 6 of Shin [ 22 ], where the discussion focuses on Venn diagrams but also considers Euler diagrams). In summary, Euler diagrams represent relations between sets – intersection, disjointness, and so on – in a way that users can read quickly and intuitively because the spatial conditions “resemble”, in some sense, the properties they represent. Although a circle does not have any literal resemblance to the abstract notion of a set, a circle encloses a region of space and any point is either inside or outside of that region, just as any object is either inside or outside of a set. Thus, placing two circles so that they overlap or are disjoint leads the viewer to the obvious inferences about the relationship between the sets in question. Similarly, the fact that SD represents the existence of an element in a set by placing a spider foot in the region of the diagram that represents that set is well matched to meaning, and allows for intuitive understanding. Hammer and Shin [ 11 ] noted that the additions to Euler’s original notation do not always provoke the same natural associations in the reader. Shading, for instance, first introduced by Venn, bears no resemblance to the emptiness of a set and has a purely conventional meaning (apart from a slightly tenuous connection between shading, darkness and the emptiness of a void). Similarly, although spiders with a single foot may be well matched to meaning, the meaning of spiders with several feet is purely conventional. Thus, our first approach to understanding the intuitive power of diagrams might be to consider a spectrum from resemblance to conventionality. However, Shin [ 22 ] argues that resemblance is not, in fact, inversely proportional to conventionality. Two cognitive properties of diagrams which are inversely proportionate to each other, however, are conventionality and the use of perceptual inferences. That is, the less a notation relies on convention, the more perceptual inferences are introduced. Furthermore, she points out that several of the conditions we might want to represent are incapable of depiction without convention, particularly disjunctive and negative information. It is not possible to depict a situation that resembles the ones described by the formulae A ∨ B and ¬A. So, any graphical notation that conveys this type of information must do so by importing symbolic features. In SD, disjunction is shown using the symbolic device of spiders’ legs and, in the full system, the logical symbol ∨. SD can depict some negative information by resemblance but not all. For instance, we can depict the situation reflected by the formula ∃x(¬A(x)) by placing a spider outside of a curve labelled A, but to show ¬∃x(A(x)), we must use the symbolic device of shading.

As well as inheriting the benefits of Euler diagrams, SD inherits some limitations: an Euler diagram can quickly become cluttered [ 15 ]. The diagram in figure 2 shows all possible intersections of four curves. We can see that this diagram has lost some of the readability of simpler examples, and the problem escalates quickly with the addition of more curves. If we want to add a new curve, say A, to figure 2 without adding any new information, we must do so such that A intersects every region, resulting in a diagram which is very difficult to draw and understand. The same problem applies to symbolic logic, however. A sentence from first-order logic that contains four or more predicate symbols or, worse, four or more variables, could also take considerable effort to read. Thus, SD, and visual logics generally, are not alone in suffering from clutter. There are ways of reducing this clutter, but these means, such as the use of discontinuous curves or overlapping edges [ 26 ], do so at the cost of some of the intuitive properties of Euler diagrams.

As a final observation about the diagram in figure 2, we note that we could produce this diagram from 1 by discarding three pieces of information about the disjointness of sets, as well as making other changes. Discarding this information requires us to add syntax to the diagram (the region representing Bird ∩ Plane), and makes the resulting diagram harder to read as a result.

In summary, spider diagrams, at least in the case of unitary diagrams, preserve and extend much of the effective and intuitive power of the underlying Euler notation. However, the rapid accumulation of regions in a diagram can mean that SD doesn’t scale well, although this lack of scalability also affects symbolic languages.

Existential graphs

Existential graphs were introduced by Peirce [ 17 ] at the end of the 19th century, at the same time as his seminal work on symbolic systems. There are two variations of the notation, α graphs and the more expressive β graphs, which are as expressive as first-order logic with equality [ 19 ]. Unlike SD, β graphs can represent predicates with any number of places. In order to make a meaningful comparison between the systems, we will consider Peirce’s β graphs with the restriction that all predicates are monadic, and call this system EG. An graph in EG is composed of predicate symbols, lines of identity (LIs) and cuts. A predicate symbol is a label representing a predicate; since our predicates are monadic we can equally well consider the labels to represent sets as predicates. An LI is a network of lines which may have any number of branches and which represents an individual (there is an exception to this rule, which we explain below). A cut is a closed curve drawn on the diagram which represents the negation of that information inside it. The final syntactic device is juxtaposition: placing graphs G1 and G2 next to each other creates a new graph whose meaning is the conjunction of the meanings of G1 and G2. Figure 3 shows an existential graph. The parts of the graph labelled G1 to G5 we call subgraphs (note that these labels are added for convenience and are not part of the notation). The subgraph G1 has one LI, three predicate labels and four cuts.

Interpreting existential graphs has often been seen as a difficult task, and this is one of the main points of criticism of the system. Shin [ 23 ] proposed a new reading procedure which is both more regular than earlier procedures and which exploits the diagrammatic properties of existential graphs. However, the reading procedure which is arguably easiest to describe informally is the endoporeutic reading proposed by Peirce himself. Informally, we read a graph from the “outside in”, or from the region of the graph which is least enclosed by cuts towards the most enclosed part. Thus, reading subgraph G2 in figure 3, we first encounter a cut, so we know that some piece of information is to be negated. Next, we encounter an LI, so we know that a statement concerning some individual will be made. Finally, the predicate labels Bird and Sman are at the ends of the LI, and we construct the meaning “it is not the case that there is some thing for which Bird and Sman are true”, or “nothing is a bird and a Superman”. More formally, we construct the formula ¬∃x(Bird (x)∧Sman(x)). So, the subgraph G2 in figure 3 conveys a subset of the information given in figure 1 by the placement of the curves labelled Bird and Sman. The conjunction of the meanings of the subgraphs G2, G3 and G4 provides the same information as the placement of curves in figure 1. Reading G1 from the outside in, we encounter an LI, denoting an individual, say x, so the constructed meaning begins with the fragment ∃x . . .. Next, we encounter a cut, and so we have ∃x¬(. . .). Next, we encounter three nested cuts, and we construct the fragment

∃x¬(¬(. . .) ∧ ¬(. . .) ∧ ¬(. . .)).

Inside these cuts are predicate labels attached to the ends of the LI, and we have ∃x¬(¬(Bird (x)) ∧ ¬(Plane(x)) ∧ ¬(Sman(x))).

Shin’s reading gives the equivalent but neater formula ∃x(Bird (x) ∨ Plane(x) ∨ Sman(x)). Thus, subgraph G1, figure 3, demonstrates how disjunction is conveyed in EG. Compare this to figure 1, where the spider conveys the same information.

Subgraph G5, figure 3, tells us that it is not the case that there are two things, say x and y, for which Sman is true and where x = y. That is, there is at most one Superman. This is the same information as is conveyed in figure 1 by the combination of the shading and spider foot in the curve labelled Sman. Inequality between objects in EG is shown by an LI which crosses an otherwise empty cut, indicating that the two (or more) extremities of the LI do not represent the same object; this is the exception to the way we read an LI mentioned above.

Thus, the spider diagram in figure 1 expresses the same meaning as the existential graph in figure 3. We note that figure 1 is much more compact than figure 3, demonstrating the expressive power of Euler diagrams and showing that, in this case, SD preserves and extends that power. Recall that spider diagrams assert information by the use of spiders, shading and the relative placement of curves. Figure 1 includes one spider, one shaded region and three curves placed so as not to intersect every region: five pieces of information. An existential graph representing the same information must include five subgraphs. Given a spider diagram, d, with n regions which are shaded or not represented in d, an existential graph, G, requires n − 1 subgraphs to represent the same information. The mapping between the spiders in d and subgraphs in G is not so straightforward, since several spiders which each have a single foot and which are placed in the same region can be represented by a single LI which crosses an other wise empty cut. The subgraphs of G will contain duplicated predicate labels, as is the case with figure 3, and distinct LIs which may refer to the same individual, whereas this duplication is not necessary in SD. This is one sense in which SD is more compact and elegant than EG. Like SD, the syntax of EG contains conventional or symbolic features: notably, cut bears no resemblance to negation. As discussed previously, however, any notation that represents negation or disjunction must do so symbolically. In comparison with Euler diagrams, it may seem that EG uses only symbolic features, with the exception of the LI. Even in this case, in which it seems reasonable to say that a line resembles, in some sense, an individual, the effect is marred by the special case of an LI that passes through an otherwise empty cut.

Similarly to the relationship between figures 1 and 3, the diagram in figure 4 has an equivalent meaning to the spider diagram in figure 2: informally, there is something which is either a bird, a plane, or is Superman and is wearing a cape. In this case, comparing the two diagrams leads us to the conclusion that EG is relatively effective in this context, since the intuitive properties of SD are hampered in figure 2 by clutter. In order to avoid asserting information about the relationship between two sets, S1 and S2, a spider diagram must include an unshaded region which contains no spiders and which represents S1 ∩ S2. In figure 2, this leads to a diagram which is difficult to read (and draw). In EG, there is no need to avoid making claims about the relationship between S1 and S2. Adding more curves to the spider diagram would render it very difficult to read, whereas adding another predicate symbol to the existential graph in figure 4 would not have that effect. Furthermore, at the end of section 2 we noted that spider diagrams may require more syntax to represent less information. This is not the case with EG: the graph in figure 4 can be produced from the graph in figure 3 by adding one piece of information (that if a particular individual is Superman, it is also wearing a cape) and discarding three pieces of information regarding the relationships between the predicates (given by subgraphs G2 to G4 in figure 4). Discarding this information results in less syntax appearing in the graph in figure 4.

Remarkably, EG does not introduce specialised syntax to represent disjunction, conjunction or implication: all these properties are represented as a byproduct of the spatial relations of predicate symbols, LIs, cuts and juxtaposition. Despite the fact that EG was almost entirely ignored by logicians until the 1960s, Peirce considered his graphical system superior to his own symbolic system [ 17 ]. Peirce categorised diagrammatic features as icons, indices and symbols. An icon represents something by its resemblance to that thing. An index represents something by “pointing it out”, much as a signpost does. A symbol represents some fact or condition merely by convention. Peirce’s aim was to create a system which was as iconic as possible. As discussed, this effort can be considered successful with respect to lines of identity, which “resemble” individual identity, but the use of cut is symbolic. Furthermore, the regularity of using LIs to represent individuals is disturbed by the special case of LIs which cross an empty cut, in which case a single LI represents two or more objects which are not the same. In contrast, the symbolic device of shading used in SD seems to us to pose less of a cognitive challenge, since the shading is deployed within the relatively iconic context of Euler circles.

Before describing Coppin’s framework in the next section, we note a final important syntactic similarity between SD and EG. Both systems feature node-link diagrams, which are lines of identity and spiders respectively. In EG the edges of the node-link diagrams represent identity and do so iconically, or by resemblance, while the nodes represent predicates, and do so symbolically. In SD the edges of the node-link diagrams represent disjunction and do so symbolically, whilst the nodes provide an iconic representation of individuals. In a recent eye tracking study [ 4 ], Burch et al. investigate the impact of the orientation and format of node-link diagrams on comprehension tasks; amongst other things, their results show that readers of node-link diagrams prioritise giving attention to nodes over links – these are the information-rich parts of the diagram. Thus, when comprehending the node-link component of a spider diagram, attention is first paid to a series of assertions about objects, making this an object-centric notation. When comprehending an existential graph, information about predicates is given prominence for the same reason, making EG predicate-centric. For our purposes, the relative efficiency of representation is of less importance than this bias towards objects (SD) or predicates (EG), accompanied by the fact that the nodal information is represented iconically by SD and symbolically by EG. 4

A framework for affordances

In this section we describe a framework that explains the fact that the differing approaches of SD and EG may each be more effective than the other in a given context.3 4.1

Pictorial and Symbolized Information

Our aim is to work towards an understanding of perception-recognition that can be used to distinguish pictorial and symbolized information. Throughout this development, it will be important to remember that perception of visual representations necessarily and simultaneously always involves both pictorial and symbolized information to various degrees. At the core of the argument is the 3 The majority of the ideas in this section are attributed to author Coppin and will form part of his forthcoming PhD thesis [ 6 ]. claim that these two types of information closely correspond to two aspects of perception-recognition that we categorise as emulation and simulation. Indeed, it is via their relative engagements of these two aspects that we will be able to distinguish between symbolic and pictorial visual representations: pictorial representations engage relatively more of the aspects or perception that we characterize as emulation, while symbolic representations (that is, representations which contain relatively more symbolic information) engage relatively more of the aspects of perception we characterize as simulation. In order to proceed, we need to establish what is involved in these two key aspects of perception.

What we refer to as emulation can loosely be described as the aspect of perception-recognition that is most closely coupled with the proximal stimuli and sensations that impinge upon an organism. With respect to vision, emulation would include the near isomorphic response of retinal receptors to the aspects of the optic array [ 9 ] to which they are specifically tuned to respond. As information gets further from this “surface interface” and is processed by higher level aspects of the perceptual-cognitive system, it becomes less accurate to characterize the process as emulation. The key characteristic of this aspect of perception-recognition is that there is a structural relationship between the organism’s response and the proximal stimulus (change) to which that response is a reaction.

What we refer to as simulation is alluded to by various terms in the cognitive science literature, such as“filling in” [ 18 ] and “prediction” [ 3 ]. This is the aspect of perception that allows us to see distal things as three-dimensional objects, even when only some subset of two-dimensional surfaces are reflecting light to our eyes. In order to achieve this, our visual systems must be able to simulate things and events in the world, in some spatio-temporal sense. This has been shown to rely on experience/memory and learning [ 12 ]. Because of this, the range of possibilities for a simulation that is a response or reaction to an external change or variation is greater than for the emulated aspects. Unlike emulation, because structural correspondence (between the proximal stimuli/change and the reaction) is not a defining characteristic of simulation, it is not easy or even possible to directly map back from the reaction to the structure of the stimuli. Emerging from all of this, the key characteristic of simulation is (subjective) extrapolation from the proximal structure of stimuli to the recognition of the distal structure of the world.

As described above, the distinction between these complementary modes of perception-recognition is noted at several points in the literature, and the modes are given various names. Hurford [ 14 ] identifies the modes with dorsal and ventral neural pathways, respectively. He notes the precedence of the dorsal/emulated mode and argues for the importance of this on the sense-making activities of our pre-linguistic ancestors and, ultimately, on our own development of language. Dorsal pathways make an initial categorisation of a stimulus that Hurford likens to a series of evaluations of predicate statements, whereas ventral pathways supply further environmental detail used to locate the stimulus in context. Thus, we use emulation to learn what a stimulus represents, before using simulation to learn where it is, how it stands in relation to its environmental context, and so on. To borrow terms from philosophy, we discover quiddity, the whatness or initial categorisation of a stimulus, then haecceity, the thisness or refined categorisation4.

At this point, an example is required. Figure 5 shows, from left to right, a realistic picture of an apple, successively less realistic depictions, the logo of the Apple company, and finally the (purely symbolic) name of that company. Emulation occurs when the viewer recognises that the apple picture on the far left is structurally similar to the emulation that occurs when looking at a real apple. Meanwhile, the activity that occurs when the viewer sees the Apple logo is less easily mapped back onto proximal stimuli and therefore more in the realm of simulation. Furthermore, the type and degree of learning required for perceivingrecognizing the Apple logo is greater than and different in quality from that required for perceiving-recognizing a photograph of an Apple. Together these two perceptual-cognitive distinctions justify distinguishing between the two representations such that we label one as being more pictorial and the other as being more symbolic.

Figure 6 presents these distinctions in grid form, and shows the mixture of emulation and simulation required to process content in a heterogeneous notation. To relate this to earlier sections, both SD and EG may be considered heterogeneous systems from this point of view, containing relatively pictorial and relatively symbolic elements. A good example of this is the differing semantics assigned to closed curves in each notation. In SD, the use of curves is strongly pictorial/emulated, whilst in EG it is strongly symbolic/emulated. Conversely, disjunction is symbolic/emulated in both notations. In particular, the node-link diagrams that feature in each notation are heterogeneous, and each notation mixes pictorial and symbolic information in opposite ways. In SD, nodes, which are spiders’ feet, use the pictorial device of placement within a region, whilst edges, spiders’ legs, are symbolic. In EG, nodes, which are predicate labels, are purely symbolic, whilst edges, lines of identity, are relatively pictorial. 4 In Coppin’s thesis the roles of memory and recognition are developed extensively in this context and these processes are posited as intermediaries between the two modes. The Emulative and simulative modes are engaged, therefore, when processing node-link diagrams in either notation. Because priority is given to the nodes of those diagrams, however, the order in which the cognitive modes are engaged differs. 4.2

Affordances of Graphic Representation Types

The framework enables predictions regarding the perceptual-cognitive affordances of the graphic representation types described in this paper. We build on the idea that capabilities for emulation and simulation share a common, and limited, resource: attention and working memory [ 1, 8, 16 ]. Presented at a high level, the predictions we make are as follows. 1. Pictured object relations or attributes interfere with, or hinder, mental simulation of object relations or attributes, intended by an author. 2. Pictured object relations or attributes can support a mental emulation intended by an author. 3. Combinations of pictured and symbolized information can (a) free resources for mental simulation of symbolized objects, and (b) symbolized information affords mental simulations that are difficult, or impossible, to emulate.

We will first consider item 1. The “free rides” provided by Euler diagrams (and by other notations), noted by several authors and named by Shimojima [ 21 ], occur when information arises in a diagram as a by-product of its syntax. In figure 7, the Euler diagram on the left tells us that all jets are planes, and no birds are planes. We can read immediately from this, as a free ride, that no jets are birds. Free rides accumulate: we can produce the diagram on the right of figure 7 by adding the curve labelled Sings to the diagram on the left; as well as adding the information that everything that sings is a bird we also learn, as a free ride, that no planes can sing and neither can any jets sing.

Free rides are a powerful component of the effectiveness of a graphical notation. They come about when a notation is well matched to meaning, allowing the viewer to use their intuition to make valid inferences for themselves. They also depend on a situation where a single piece of relatively pictorial syntax depicts several things at once, and this can become a significant distraction when comprehending a cluttered diagram. This point is related to the authors’ previous work on constraint diagrams [ 7 ], in which we argue that generalized constraint diagrams possess the advantage over the original constraint diagram notation of being less diffuse; constructing the meaning of an individual piece of syntax can be done with reference to relatively fewer diagrammatic elements. However, the features that make generalized constraint diagrams less diffuse may also reduce the number of free rides available.

To consider cases in the current context where free rides may be counterproductive, recall that figures 2 and 4 display equivalent information. In figure 2, a relatively pictorial device is used to depict sets but since every possible set intersection is depicted, no information about the sets is conveyed by the curves in isolation. The informational content of the diagram relates to upper and lower bounds on set cardinality, and includes disjunctive information. This in formation is provided by shading and a spider, although the interpretation of the information depends on the relative position of curves. In the context of Coppin’s framework, processing this information requires, predominately, simulation, and the framework predicts that a less pictorial approach may be effective. This is stated as item 1 above: pictured object relations or attributes interfere with mental simulation. In figure 4, we saw that EG conveys the same information as SD, figure 2, and we claimed that it did so relatively effectively. The framework predicts this effect, since in figure 4 the information requiring simulation is conveyed by predominately symbolic means.

On the other hand, consider figures 1 and 3. In figure 1, several pieces of information about relations between sets are conveyed simultaneously: no birds are planes, no birds are superman, and so on. Although this spider diagram also includes disjunctive information, the majority of the content is conveyed via the spatial relations of curves, and the viewer benefits from a natural mapping from these relations to relations between sets. The process of comprehending figure 1 is relatively emulative. We can see that a pictorial approach is effective by considering an existential graph with equivalent meaning to figure 1, shown in figure 3. In this figure no free rides occur and so each set relation is given explicitly using a symbolic representation. This is an example of the effect we state in item 2 above: pictured object relations or attributes can support emulation.

Figure 8 shows a spider diagram in which “uncertainty”, or disjunctive information, is removed from figure 1 whilst figure 9 shows an existential graph with an equivalent meaning. The information conveyed by figures 8 and 9 is emulated, consisting of a series of initial categorisations: there are sets of birds, planes and supermen, there is a superman. Comparing the two representations shows that the pictorial approach of SD is certainly less cluttered and, we believe, more effective.

Fig. 8. It’s Superman!

Fig. 9. Symbolic representation of emulated content.

Items 3a and 3b in our list of assertions on page 57 can be seen as corollaries to items 1 and 2. As we have seen, although the curves used in SD (and in Euler diagrams) to represent sets or predicates are more pictorial than the predicate labels of EG, the use of curves can reduce the effectiveness of the notation by causing clutter. This calls to mind Peirce’s stated goal for EG ([ 17 ], quoted in Shin [ 23 ]) that a diagram should be “as iconic as possible”: perhaps we should add to this the caveat “but not more”. When something cannot be depicted, an approach that represents that information using a relatively symbolic device may be easier to comprehend and more scalable than one that uses a metaphor of resemblance. 5

Conclusion

Our comparison of the affordances of SD and EG is not undertaken in order to conclude which system is the most effective. Instead, we have shown that the interaction of pictorial and symbolic features can promote or hinder certain cognitive processes, which we call emulation and simulation. Using the framework, we have explained the fact there are certain tasks for which EG is surprisingly effective, although EG is (we believe) more cumbersome and less intuitive than SD. We have considered static comprehension tasks only, but SD and EG are reasoning systems. In further work, we intend to use the framework to evaluate the two notations when used to construct and comprehend proofs, and to conduct empirical studies which test the validity of the findings.

A more finely grained version of the predictions in section 4.2 is part of Coppin’s thesis. Using these predictions in a consideration of emulated and simulated features in existing notations could lead to a principled approach to generating effective diagrams. The same problem is addressed, though using quite different means to our own, by Rodgers et al. [ 20 ] in their definition of well-formedness criteria for Euler diagrams and the effect of the criteria on readability. The fine grained principles could also be used by designers of new notations, through a consideration of the informational domain of the notation and the cognitive processes implied.

We also believe the framework can be used to investigate “layers” of information within graphical notations. As we have discussed, both SD and EG include node-link diagrams, and we believe the balance of pictorial/symbolic information at the nodes of these diagrams must be appropriate to the task in question. We conjecture that the node-link diagrams form part of an upper layer or “cognitive foreground” of the notations. Both SD and EG have a series of nested curves as a “background” layer, though only SD has a background layer which can be interpreted independently of other diagrammatic content. We intend to study the existence of layers of content and the effect of their degrees of independent coherence by conducting eye tracking studies that investigate the ways in which users pay attention to the syntactic elements of diagrams that include node-link diagrams amongst other syntax.

Acknowledgements: The authors would like to thank Steve Hockema and Gem Stapleton for helpful discussions around the topics in this paper.

References

1. A. Baddeley . Working memory . Comptes Rendus de l' Acad´emie des Sciences-Series III-Sciences de la Vie , 321 ( 2-3 ): 167 - 173 , 1998 .

2. A. Blackwell and T. R. Green . Notational systems - the cognitive dimensions of notations framework . In John M. Carroll, editor, HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science , Interactive Technologies, chapter 5 , pages 103 + . Morgan Kaufmann, San Francisco, CA, USA, 2003 .

3. A. Bubic , D. Y. Von Cramon , and R. I. Schubotz . Prediction, cognition and the brain . Frontiers in human neuroscience, 4 , 2010 .

4. M. Burch , N. Konevtsova , J. Heinrich , M. Hoeferlin , and D. Weiskopf . Evaluation of traditional, orthogonal, and radial tree diagrams by an eye tracking study . Visualization and Computer Graphics , IEEE Transactions on, 17 ( 12 ): 2440 - 2448 , December 2011 .

5. M. Chein and M. Mugnier . Conceptual graphs: fundamental notions . Revue d´e Intelligence Artificielle , 6 : 365 - 406 , 1992 .

6. P. Coppin . Perceptual-cognitive properties of realistic pictures, outline drawings, diagrams, and sentences: toward a science of information design . PhD thesis , University of Toronto, Available in 2012 .

7. P. Coppin , J. Burton , and S. Hockema . An attention based theory to explore affordances of textual and diagrammatic proofs . In Ashok Goel, Mateja Jamnik, and N. Narayanan, editors, Diagrammatic Representation and Inference , volume 6170 of Lecture Notes in Computer Science, chapter 27 , pages 271 - 278 -278. Springer Berlin / Heidelberg, Berlin, Heidelberg, 2010 .

8. N. Cowan . An embedded-processes model of working memory. Models of working memory: Mechanisms of active maintenance and executive control , pages 62 - 101 , 1999 .

9. J. J. Gibson . The ecological approach to visual perception . Lawrence Erlbaum , 1986 .

10. C. Gurr and K. Tourlas . Towards the principled design of software engineering diagrams . In Proceedings of 22nd International Conference on Software Engineering , pages 509 - 518 . ACM Press, 2000 .

11. E. Hammer and S. J. Shin . Euler's visual logic . History and Philosophy of Logic , pages 1 - 29 , 1998 .

12. S. A. Hockema . Perception as prediction . PhD thesis , Indiana University, 2004 .

13. J. Howse , F. Molina , J. Taylor , S. Kent, and J. Gil . Spider diagrams: A diagrammatic reasoning system . Journal of Visual Languages and Computing , 12 ( 3 ): 299 - 324 , June 2001 .

14. J. R. Hurford . The neural basis of Predicate-Argument structure . Behavioral and Brain Sciences , 23 ( 6 ), 2003 .

15. C. John , A. Fish , J. Howse , and J. Taylor . Exploring the notion of clutter in Euler diagrams . In 4th International Conference on the Theory and Application of Diagrams , pages 267 - 282 , Stanford, USA, 2006 . Springer.

16. G. A. Miller . The magical number seven, plus or minus two: some limits on our capacity for processing information . Psychological review , 63 ( 2 ): 81 , 1956 .

17. C. S. Peirce . Collected Papers , volume 4 . Harvard University Press, 1933 .

18. L. Pessoa , E. Thompson , A. No¨e, and Others. Finding out about filling-in: A guide to perceptual completion for visual science and the philosophy of perception . Behavioral and Brain Sciences , 21 ( 6 ): 723 - 748 , 1998 .

19. D. D. Roberts . The Existential Graphs of Charles S. Peirce. Mouton , 1973 .

20. P. Rodgers , L. Zhang , G. Stapleton, and A. Fish . Embedding wellformed euler diagrams . In Thirteenth International Conference on Information Visualization , pages 585 - 593 . IEEE, 2008 .

21. A. Shimojima . Inferential and expressive capacities of graphical representations: Survey and some generalizations . In 3rd International Conference on the Theory and Application of Diagrams , volume 2980 of LNAI , pages 18 - 21 , Cambridge, UK, 2004 . Springer.

22. S. J. Shin . The Logical Status of Diagrams. CUP , 1994 .

23. S. J. Shin . The Iconic Logic of Peirce's Graphs . Bradford Book , 2002 .

24. G. Stapleton, J. Howse , and J. Taylor . A decidable constraint diagram reasoning system . Journal of Logic and Computation , 15 ( 6 ): 975 - 1008 , December 2005 .

25. G. Stapleton, S. Thompson , J. Howse , and J. Taylor. The expressiveness of spider diagrams . Journal of Logic and Computation , 14 ( 6 ): 857 - 880 , December 2004 .

26. G. Stapleton, L. Zhang , J. Howse, and P. Rodgers . Drawing euler diagrams with circles: The theory of piercings . IEEE transactions on visualization and computer graphics , September 2010 .