Introduction

Sense-making refers to the process by which we give meaning to our experiences [1,2]. This process should be thought of as an agent with a specific physical body acting within, and experiencing, a physical environment. In this view, meaning emerges through this very interaction and experience. Our goal is to model this sense-making process formally. Diagrammatic reasoning is an apt domain for the formalisation of sense-making, because both the geometrical syntax and the intended meaning (semantics) of diagrams can be precisely characterised. Following a sense-making approach, we put forward that no diagram is meaningful by itself, but that diagrams prompt a user to give meaning and reason with them in an active, embodied manner.

To illustrate our approach, we will use the particular example of a Hasse diagram (Fig. 1; top-left). The diagram represents a partially ordered set (poset) as follows: 𝑥 is covered by 𝑦 (𝑥 ≺ 𝑦) whenever point 𝑥 is lower than point 𝑦, and 𝑥 and 𝑦 are connected by a line in the diagram. Two of the possible ways for a user to make sense of, for instance, points 𝑐, 𝑒 and ℎ, and the lines that connect them, in Fig. 1, are that (a) point 𝑐 with 𝑒, and 𝑒 with ℎ, form two pairs of entities that are linked by lines, and (b) points 𝑐, 𝑒, and ℎ are locations on a downward path with direction from 𝑐 to 𝑒. This understanding of the geometric configuration allows for the emergence of inferences such as the following: Since there is a linked path from 𝑐 to 𝑒 and from 𝑒 to ℎ, then there is a linked path from 𝑐 to ℎ. Because this is a downward path from 𝑐 to ℎ, we infer that 𝑐 is greater than ℎ.

We claim that image schemas are useful for modeling this embodied sense-making process, because they comprise cognitive structures abstracting repeated sensorimotor contingencies like container, support, verticality and balance [3,4]. Since image schemas are Gestalts, they can guide perception and inference by being systematically integrated with our experience. This way, they structure them into blended concepts where novel structure, and thus novel meaning, can emerge. This process is called conceptual blending [5]. Our proposal in the context of diagrammatic reasoning is that the geometry of a diagram and the diagram as it is made sense of by a user are distinct. Sense-making cognitively structures the diagram in a way that is more meaningful for a user than pure geometry. We model this structuring process by considering image schemas and subparts of the geometric configuration of a diagram as constituents of a network of conceptual blends representing the process of making sense of the diagram [6].

Approach

Sense-making is defined within the scope of enactive cognition as the process of an autonomous agent bringing its own original meaning upon its environment [1,2]. Analogously, the meaning of a diagram also emerges during a constructive and imaginative process on the part of the user. [7]. Image schemas are fundamental for such a process, because they organise and structure our experience, and thus guide our reasoning, by integrating their internal structure with what we perceive and experience. This integration can be described according to the principles of conceptual blending. Conceptual blending is a process of human cognition that operates on mental spaces; "small conceptual packets constructed as we think and talk" [5, p. 40]. Though blending, particular elements and relations of originally separate input mental spaces are combined into a blended space, in which new elements and relations emerge [5].

To model the sense-making of diagrams we construct first-order logic (FOL) theories of their geometry, as well as of the image schemas involved. For the diagram geometry, we draw from Qualitative Spatial Reasoning (QSR) formalisms to characterise spatial entities in terms of their topological relations, shape, and relative position. Concretely, we have used the Common Algebraic Specification Language (CASL) [8] and the HEterogenous ToolSet (HETS) [9] to construct and verify logical theories of image schemas, namely, of link, source-path-goal, verticality, scale, container, covering and surface (see [6] for the first four). Using the same tools, we have implemented the blend networks modeling the sense-making of examples of Hasse, Euler, Concept and Entity-Relationship diagrams [10].

The formal approach we adopt for blending is that of amalgams [11]. Very briefly, the input spaces, the generic space, and the blend, are all taken to be members of a set partially ordered by a generalisation relation. There is a lattice of possible generalisations of the input spaces, which remove some information, and blend the new, generalised input spaces. However, blending more generalised input spaces gives rise to blends with lower specificity (less information). Generalising the description of amalgams to a category-theoretical view allows us to discuss blending independently of the specific representation language in which the inputs are expressed [12].

Having the aforementioned formal tools, we can construct networks of blends that integrate several image schemas with a geometric configuration, reflecting particular reasoning tasks. In the case of the Hasse diagram, we propose that the schemas involved in various reasoning tasks are link, source-path-goal, scale and verticality. All input spaces (image schemas and geometry) are involved in an intricate network of correspondences and blends, and so they all structure each other, giving rise to blended spaces that are both geometric and image-schematic at once. Inference emerges within this network in its entirety. The correspondences between various instances of these schemas and substructures of the Hasse configuration can yield the Hasse diagram as comprising several paths of linked points, arranged at several levels of generality along an upward vertical axis. For example, in Fig. 1 we show the (simplified) network of blends wherein the inference that 𝑐 > ℎ emerges, as walking along a downward path of connected locations from 𝑐 to 𝑒 and then to ℎ. Other inferences, possible through the involvement of the verticality and scale schemas, are that some elements of the poset are on the same level of generality, e.g., 𝑒, 𝑓 and 𝑔, and, finally, ℎ is understood to be the least element of the poset, while 𝑎 the greatest. Similarly, in diagrams with closed curves, such as Venn and Euler diagrams (for the latter, see Fig. 2), correspondences can be built between the container schema and the geometry of the diagram. Namely, the boundary, inside and outside of the container schema is put in correspondence with the curve itself, the geometrical area inside it, and the geometrical area outside it, respectively. This allows making sense of a closed curve as a container, and reasoning about the diagram. For example, in Fig. 2 we see the blending network modeling how we make sense of closed curve 𝐵 as a container. Constructing more correspondences would allow us, for instance, to infer that closed curve 𝐴 is inside closed curve 𝐵, and many other facts.

Related Work

The sense-making process as such is seldom addressed in AI [13]. However, some initial work has employed image schemas to model sense-making. Falomir and Plaza blend image schemas with a QSR description of an icon, in order to make sense of the latter [14]. This conceptual work has greatly inspired our research. Embodied Construction Grammars allow for the formalisation and implementation of language understanding by putting in correspondence components of specific schemas (image schemas, and others) to phonemes [15,16].

There exist also some proposals to formalise image schemas and the relations among them. Rodriguez and Egenhofer provide a relational algebra inspired by the container and surface schemas, used to model, and reason about, spatial relations of objects in an indoor scene [17]. Image schemas have also been used to model planning and actions of agents, by recursively defining some image schemas in terms of others [18]. In both these works, the formalisations are merely inspired by image schemas, rather than faithful representations of their descriptions in the literature. Kuhn formalised image schemas, and their combinations, as ontology relations using functional programming [19]. In a recent, comprehensive work, Hedblom modelled image schemas as families of interrelated logical theories, with each schema comprising a combination of primitive components, and using QSR, and other, formalisms that capture the spatiotemporal content of schemas [20]. In our approach, we chose not to use such formalisms to capture the internal structure of image schemas, and not to predefine specific variants or combinations of image schemas. Rather, our aim was to allow aspects of the image schemas (for example, the number of locations on a path, or the number of levels on a scale) to be shaped partly by their correspondences with the geometry of the diagram [21].

As for diagrammatic reasoning, some literature posits that the effectiveness of diagrams lies in the fact that certain visual relations in the geometry of the diagram align with certain information in the domain of reference of the diagram, enabling directly observing additional information from the geometry, without needing additional inference steps [22,23]. Here, we expanded in this direction by modeling the origin of these properties as the blending of image schemas with the geometry of a diagram.

Discussion and Conclusions

The predominant logical approaches to diagrammatic reasoning do not take into account the user as an embodied being, actively reasoning with the diagram. We thus believe sense-making with image schemas provides insight into the cognitive aspects of diagrammatic reasoning and so we set out to model it. We have presented a formal framework of the sense-making of diagrams, modeling the way users blend their embodied cognitive structures with the geometry of a diagram, capturing the emergence of inferences [6]. To the best of our knowledge, our approach is a novel contribution to the literature of both image schemas and diagrammatic reasoning. Below we describe the directions towards which we are currently extending our work.

We are interested in using our framework to investigate why some diagrammatic formalisms are more effective than others for reasoning. In principle, almost any image schema and any diagram can take part in a network of conceptual blends, with different inferences emerging. However, some of these inferences may not be valid given the semantics of a particular diagram. Therefore, a diagram would be effective if there exist some networks of blends of its geometry and certain image schemas, wherein valid inferences about its semantics can emerge. Another way to evaluate a blend network, and thus the effectiveness of a diagram, is the formalisation and use of governing principles for conceptual blending. These principles assess features such as how tightly integrated a blend is, how much of the structure of the input spaces it incorporates, and whether the correspondences between elements of the input spaces appear in the blend [5, ch. 16]. These principles are proposed to show how cognitively useful a blending network is, and our proposal would be that a diagram 𝐷 1 is more effective than diagram 𝐷 2 if its geometry can be blended with image schemas in a blending network that satisfies the governing principles more than the corresponding blending network for 𝐷 2 . This work could provide guidelines for the design of effective diagrammatic and graphical visualisations by characterising diagrammatic formalisms as effective in a manner that takes into account our embodiment.

Going in a different, but related, direction, we also plan to implement a pipeline that explores possible blends. This way we could model the sense-making of diagrammatic and other visual formalisms, discovering various possible senses of them in a cognitively-inspired way. However, this may prove to be challenging because the resulting search space is vast [24]. The governing principles can serve as heuristics to guide our search of blends between image schemas and diagrams. Our framework can be validated in part by applying it to diagrams used in existing experimental studies on diagrammatic formalisms and their features, and how they affect reasoning speed and accuracy. It is possible to check if the blending networks selected by our pipeline correspond to diagrams that had high effectiveness (high reasoning speed and accuracy) in behavioral experiments.

Finally, although so far we have modelled a blending process to represent the sense-making of geometric configurations of diagrams, our model could be applied in other domains. Furthermore, the entire framework could eventually be generalised in a representation-independent manner as described in [12]. We believe our work has potential as a general-purpose, parsimonious module for modeling the way an agent makes sense of its environment, using a few preexisting structures, i.e., image schemas, together with a search and blending process. Image schemas, or similar concepts, have indeed been proposed as useful primitives for common-sense reasoning [25,26,27].