We present our approach to modeling the sense-making of diagrams as networks of conceptual blends that structure the geometric configuration of a diagram with embodied image schemas. We believe the types of inferences we confer to diagrammatic representations emerge as we cognitively construct these networks, and such inferences can be obtained in our model. We formalise image schemas and the geometric configurations of diagrams as FOL theories, drawing from Qualitative Spatial Reasoning formalisms; blends of image schemas with geometric configurations are then computed based on the theory of amalgams. We argue that this approach to sense-making of diagrams is more cognitively apt than the mainstream view of a diagram being a syntactic representation of some underlying logical semantics.
Sense-making refers to the process by which we give meaning to our experiences [
To illustrate our approach, we will use the particular example of a Hasse diagram (Fig. 1; top-left). The diagram represents a partially ordered set (poset) as follows: is covered by ( ≺ ) whenever point is lower than point , and and are connected by a line in the diagram. Two of the possible ways for a user to make sense of, for instance, points , and ℎ, and the lines that connect them, in Fig. 1, are that (a) point with , and with ℎ, form two pairs of entities that are linked by lines, and (b) points , , and ℎ are locations on a downward path with direction from to . This understanding of the geometric configuration allows for the emergence of inferences such as the following: Since there is a linked path from to and
VERTICALITY base
LINK #1 Entity 1 Link Entity 2
SPG #1
Loc 2
Loc 1 LINK #2
SPG #2
Loc 1
Loc 2 from to ℎ, then there is a linked path from to ℎ. Because this is a downward path from to ℎ, we infer that is greater than ℎ.
We claim that image schemas are useful for modeling this embodied sense-making process,
because they comprise cognitive structures abstracting repeated sensorimotor contingencies
like container, support, verticality and balance [
Sense-making is defined within the scope of enactive cognition as the process of an autonomous
agent bringing its own original meaning upon its environment [
To model the sense-making of diagrams we construct first-order logic (FOL) theories of their
geometry, as well as of the image schemas involved. For the diagram geometry, we draw from
Qualitative Spatial Reasoning (QSR) formalisms to characterise spatial entities in terms of their
topological relations, shape, and relative position. Concretely, we have used the Common
Algebraic Specification Language (CASL) [
The formal approach we adopt for blending is that of amalgams [
Having the aforementioned formal tools, we can construct networks of blends that integrate several image schemas with a geometric configuration, reflecting particular reasoning tasks. In the case of the Hasse diagram, we propose that the schemas involved in various reasoning tasks are link, source-path-goal, scale and verticality. All input spaces (image schemas and geometry) are involved in an intricate network of correspondences and blends, and so they all structure each other, giving rise to blended spaces that are both geometric and image-schematic at once. Inference emerges within this network in its entirety. The correspondences between various instances of these schemas and substructures of the Hasse configuration can yield the Hasse diagram as comprising several paths of linked points, arranged at several levels of generality along an upward vertical axis. For example, in Fig. 1 we show the (simplified) network of blends wherein the inference that > ℎ emerges, as walking along a downward path of connected locations from to and then to ℎ. Other inferences, possible through the involvement of the verticality and scale schemas, are that some elements of the poset are on the same level of generality, e.g., , and , and, finally, ℎ is understood to be the least element of the poset, while the greatest.
C
A
B C
A
B Boundary
Inside Outside
Similarly, in diagrams with closed curves, such as Venn and Euler diagrams (for the latter, see Fig. 2), correspondences can be built between the container schema and the geometry of the diagram. Namely, the boundary, inside and outside of the container schema is put in correspondence with the curve itself, the geometrical area inside it, and the geometrical area outside it, respectively. This allows making sense of a closed curve as a container, and reasoning about the diagram. For example, in Fig. 2 we see the blending network modeling how we make sense of closed curve as a container. Constructing more correspondences would allow us, for instance, to infer that closed curve is inside closed curve , and many other facts.
The sense-making process as such is seldom addressed in AI [
There exist also some proposals to formalise image schemas and the relations among them. Rodriguez and Egenhofer provide a relational algebra inspired by the container and surface schemas, used to model, and reason about, spatial relations of objects in an indoor scene [17]. Image schemas have also been used to model planning and actions of agents, by recursively defining some image schemas in terms of others [ 18]. In both these works, the formalisations are merely inspired by image schemas, rather than faithful representations of their descriptions in the literature. Kuhn formalised image schemas, and their combinations, as ontology relations using functional programming [19]. In a recent, comprehensive work, Hedblom modelled image schemas as families of interrelated logical theories, with each schema comprising a combination of primitive components, and using QSR, and other, formalisms that capture the spatiotemporal content of schemas [20]. In our approach, we chose not to use such formalisms to capture the internal structure of image schemas, and not to predefine specific variants or combinations of image schemas. Rather, our aim was to allow aspects of the image schemas (for example, the number of locations on a path, or the number of levels on a scale) to be shaped partly by their correspondences with the geometry of the diagram [21].
As for diagrammatic reasoning, some literature posits that the efectiveness of diagrams lies in the fact that certain visual relations in the geometry of the diagram align with certain information in the domain of reference of the diagram, enabling directly observing additional information from the geometry, without needing additional inference steps [22, 23]. Here, we expanded in this direction by modeling the origin of these properties as the blending of image schemas with the geometry of a diagram.
The predominant logical approaches to diagrammatic reasoning do not take into account the
user as an embodied being, actively reasoning with the diagram. We thus believe sense-making
with image schemas provides insight into the cognitive aspects of diagrammatic reasoning
and so we set out to model it. We have presented a formal framework of the sense-making of
diagrams, modeling the way users blend their embodied cognitive structures with the geometry
of a diagram, capturing the emergence of inferences [
We are interested in using our framework to investigate why some diagrammatic formalisms are more efective than others for reasoning. In principle, almost any image schema and any diagram can take part in a network of conceptual blends, with diferent inferences emerging. However, some of these inferences may not be valid given the semantics of a particular diagram. Therefore, a diagram would be efective if there exist some networks of blends of its geometry and certain image schemas, wherein valid inferences about its semantics can emerge. Another way to evaluate a blend network, and thus the efectiveness of a diagram, is the formalisation and use of governing principles for conceptual blending. These principles assess features such as how tightly integrated a blend is, how much of the structure of the input spaces it incorporates, and whether the correspondences between elements of the input spaces appear in the blend [5, ch. 16]. These principles are proposed to show how cognitively useful a blending network is, and our proposal would be that a diagram 1 is more efective than diagram 2 if its geometry can be blended with image schemas in a blending network that satisfies the governing principles more than the corresponding blending network for 2. This work could provide guidelines for the design of efective diagrammatic and graphical visualisations by characterising diagrammatic formalisms as efective in a manner that takes into account our embodiment.
Going in a diferent, but related, direction, we also plan to implement a pipeline that explores possible blends. This way we could model the sense-making of diagrammatic and other visual formalisms, discovering various possible senses of them in a cognitively-inspired way. However, this may prove to be challenging because the resulting search space is vast [24]. The governing principles can serve as heuristics to guide our search of blends between image schemas and diagrams. Our framework can be validated in part by applying it to diagrams used in existing experimental studies on diagrammatic formalisms and their features, and how they afect reasoning speed and accuracy. It is possible to check if the blending networks selected by our pipeline correspond to diagrams that had high efectiveness (high reasoning speed and accuracy) in behavioral experiments.
Finally, although so far we have modelled a blending process to represent the sense-making of
geometric configurations of diagrams, our model could be applied in other domains. Furthermore,
the entire framework could eventually be generalised in a representation-independent manner
as described in [
Springer, 2007. [20] M. M. Hedblom, Image schemas and concept invention: cognitive, logical, and linguistic investigations, Springer, 2020. [21] D. Bourou, M. Schorlemmer, E. Plaza, A cognitively-inspired model for making sense of
Hasse diagrams, in: CCIA Proc, IOS Press, 2021. [22] A. Shimojima, On the Eficacy of Representation, Ph.D. thesis, Indiana Univ, 1996. [23] G. Stapleton, M. Jamnik, A. Shimojima, What makes an efective representation of information: a formal account of observational advantages, J Logic Lang Inf 26 (2017) 143–177. [24] R. Confalonieri, M. Eppe, M. Schorlemmer, O. Kutz, R. Penaloza, E. Plaza, Upward refinement operators for conceptual blending in the description logic ℰ ℒ++, Ann Math Artif Intell 82 (2018) 69–99. [25] M. Shanahan, M. Crosby, B. Beyret, L. Cheke, Artificial intelligence and the common sense of animals, Trends Cogn Sci 24 (2020) 862–872. [26] M. Thosar, C. A. Mueller, G. Jaeger, M. Pfingsthorn, M. Beetz, S. Zug, T. Mossakowski, Substitute selection for a missing tool using robot-centric conceptual knowledge of objects, in: SAC Proc, ACM, 2020, pp. 972–979. [27] M. M. Hedblom, M. Pomarlan, R. Porzel, R. Malaka, M. Beetz, Dynamic action selection using image schema-based reasoning for robots, in: JOWO Proc, 2021.