=Paper=
{{Paper
|id=Vol-3140/bourou
|storemode=property
|title=Embodied Sense-Making of Diagrams as Conceptual Blending with Image Schemas
|pdfUrl=https://ceur-ws.org/Vol-3140/paper5.pdf
|volume=Vol-3140
|authors=Dimitra Bourou,Marco Schorlemmer,Enric Plaza
|dblpUrl=https://dblp.org/rec/conf/isd2/BourouSP22
}}
==Embodied Sense-Making of Diagrams as Conceptual Blending with Image Schemas==
https://ceur-ws.org/Vol-3140/paper5.pdf
Embodied Sense-Making of Diagrams as Conceptual
Blending with Image Schemas
Dimitra Bourou1,2 , Marco Schorlemmer1,2 and Enric Plaza1
1
Artificial Intelligence Research Institute, IIIA-CSIC, Bellaterra, Catalonia, Spain
2
Dept. Ciències de la Computació, Universitat Autònoma de Barcelona, Bellaterra, Catalonia, Spain
Abstract
We present our approach to modeling the sense-making of diagrams as networks of conceptual blends
that structure the geometric configuration of a diagram with embodied image schemas. We believe
the types of inferences we confer to diagrammatic representations emerge as we cognitively construct
these networks, and such inferences can be obtained in our model. We formalise image schemas and
the geometric configurations of diagrams as FOL theories, drawing from Qualitative Spatial Reasoning
formalisms; blends of image schemas with geometric configurations are then computed based on the
theory of amalgams. We argue that this approach to sense-making of diagrams is more cognitively apt
than the mainstream view of a diagram being a syntactic representation of some underlying logical
semantics.
Keywords
diagrammatic reasoning, sense-making, image schema, conceptual blending
1. Introduction
Sense-making refers to the process by which we give meaning to our experiences [1, 2]. This
process should be thought of as an agent with a specific physical body acting within, and experi-
encing, a physical environment. In this view, meaning emerges through this very interaction and
experience. Our goal is to model this sense-making process formally. Diagrammatic reasoning
is an apt domain for the formalisation of sense-making, because both the geometrical syntax
and the intended meaning (semantics) of diagrams can be precisely characterised. Following
a sense-making approach, we put forward that no diagram is meaningful by itself, but that
diagrams prompt a user to give meaning and reason with them in an active, embodied manner.
To illustrate our approach, we will use the particular example of a Hasse diagram (Fig. 1;
top-left). The diagram represents a partially ordered set (poset) as follows: π₯ is covered by π¦
(π₯ βΊ π¦) whenever point π₯ is lower than point π¦, and π₯ and π¦ are connected by a line in the
diagram. Two of the possible ways for a user to make sense of, for instance, points π, π and β,
and the lines that connect them, in Fig. 1, are that (a) point π with π, and π with β, form two
pairs of entities that are linked by lines, and (b) points π, π, and β are locations on a downward
path with direction from π to π. This understanding of the geometric configuration allows for
the emergence of inferences such as the following: Since there is a linked path from π to π and
The Sixth Image Schema Day (ISD6), March 24β25, 2022, JΓΆnkΓΆping University, Sweden
$ dbourou@iiia.csic.es (D. Bourou); marco@iiia.csic.es (M. Schorlemmer); enric@iiia.csic.es (E. Plaza)
Β© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
a
b c d LINK #1
Entity 1
e f g Link
Entity 2
h
SPG #1
Loc 2
Loc 1
VERTICALITY
base
LINK #2
SPG #2
Loc 1
Loc 2
Figure 1: A network of conceptual blends modeling how we can make sense of a Hasse diagram to infer
that π > β.
from π to β, then there is a linked path from π to β. Because this is a downward path from π to
β, we infer that π is greater than β.
We claim that image schemas are useful for modeling this embodied sense-making process,
because they comprise cognitive structures abstracting repeated sensorimotor contingencies
like container, support, verticality and balance [3, 4]. Since image schemas are Gestalts,
they can guide perception and inference by being systematically integrated with our experience.
This way, they structure them into blended concepts where novel structure, and thus novel
meaning, can emerge. This process is called conceptual blending [5]. Our proposal in the context
of diagrammatic reasoning is that the geometry of a diagram and the diagram as it is made sense
of by a user are distinct. Sense-making cognitively structures the diagram in a way that is more
meaningful for a user than pure geometry. We model this structuring process by considering
image schemas and subparts of the geometric configuration of a diagram as constituents of a
network of conceptual blends representing the process of making sense of the diagram [6].
2. Approach
Sense-making is defined within the scope of enactive cognition as the process of an autonomous
agent bringing its own original meaning upon its environment [1, 2]. Analogously, the meaning
of a diagram also emerges during a constructive and imaginative process on the part of the
user. [7]. Image schemas are fundamental for such a process, because they organise and
structure our experience, and thus guide our reasoning, by integrating their internal structure
with what we perceive and experience. This integration can be described according to the
principles of conceptual blending. Conceptual blending is a process of human cognition that
operates on mental spaces; βsmall conceptual packets constructed as we think and talkβ[5, p.
40]. Though blending, particular elements and relations of originally separate input mental
spaces are combined into a blended space, in which new elements and relations emerge [5].
To model the sense-making of diagrams we construct first-order logic (FOL) theories of their
geometry, as well as of the image schemas involved. For the diagram geometry, we draw from
Qualitative Spatial Reasoning (QSR) formalisms to characterise spatial entities in terms of their
topological relations, shape, and relative position. Concretely, we have used the Common
Algebraic Specification Language (CASL) [8] and the HEterogenous ToolSet (HETS) [9] to
construct and verify logical theories of image schemas, namely, of link, source-path-goal,
verticality, scale, container, covering and surface (see [6] for the first four). Using the
same tools, we have implemented the blend networks modeling the sense-making of examples
of Hasse, Euler, Concept and Entity-Relationship diagrams [10].
The formal approach we adopt for blending is that of amalgams [11]. Very briefly, the input
spaces, the generic space, and the blend, are all taken to be members of a set partially ordered by
a generalisation relation. There is a lattice of possible generalisations of the input spaces, which
remove some information, and blend the new, generalised input spaces. However, blending
more generalised input spaces gives rise to blends with lower specificity (less information).
Generalising the description of amalgams to a category-theoretical view allows us to discuss
blending independently of the specific representation language in which the inputs are expressed
[12].
Having the aforementioned formal tools, we can construct networks of blends that integrate
several image schemas with a geometric configuration, reflecting particular reasoning tasks. In
the case of the Hasse diagram, we propose that the schemas involved in various reasoning tasks
are link, source-path-goal, scale and verticality. All input spaces (image schemas and
geometry) are involved in an intricate network of correspondences and blends, and so they all
structure each other, giving rise to blended spaces that are both geometric and image-schematic
at once. Inference emerges within this network in its entirety. The correspondences between
various instances of these schemas and substructures of the Hasse configuration can yield
the Hasse diagram as comprising several paths of linked points, arranged at several levels
of generality along an upward vertical axis. For example, in Fig. 1 we show the (simplified)
network of blends wherein the inference that π > β emerges, as walking along a downward
path of connected locations from π to π and then to β. Other inferences, possible through the
involvement of the verticality and scale schemas, are that some elements of the poset are on
the same level of generality, e.g., π, π and π, and, finally, β is understood to be the least element
of the poset, while π the greatest.
CONTAINER
C B Boundary
A
Inside
Outside
C B
A
Figure 2: A network of conceptual blends modeling the sense-making of a Euler diagram.
Similarly, in diagrams with closed curves, such as Venn and Euler diagrams (for the latter,
see Fig. 2), correspondences can be built between the container schema and the geometry
of the diagram. Namely, the boundary, inside and outside of the container schema is put in
correspondence with the curve itself, the geometrical area inside it, and the geometrical area
outside it, respectively. This allows making sense of a closed curve as a container, and reasoning
about the diagram. For example, in Fig. 2 we see the blending network modeling how we make
sense of closed curve π΅ as a container. Constructing more correspondences would allow us, for
instance, to infer that closed curve π΄ is inside closed curve π΅, and many other facts.
3. Related Work
The sense-making process as such is seldom addressed in AI [13]. However, some initial work
has employed image schemas to model sense-making. Falomir and Plaza blend image schemas
with a QSR description of an icon, in order to make sense of the latter [14]. This conceptual work
has greatly inspired our research. Embodied Construction Grammars allow for the formalisation
and implementation of language understanding by putting in correspondence components of
specific schemas (image schemas, and others) to phonemes [15, 16].
There exist also some proposals to formalise image schemas and the relations among them.
Rodriguez and Egenhofer provide a relational algebra inspired by the container and surface
schemas, used to model, and reason about, spatial relations of objects in an indoor scene [17].
Image schemas have also been used to model planning and actions of agents, by recursively
defining some image schemas in terms of others [18]. In both these works, the formalisations
are merely inspired by image schemas, rather than faithful representations of their descriptions
in the literature. Kuhn formalised image schemas, and their combinations, as ontology relations
using functional programming [19]. In a recent, comprehensive work, Hedblom modelled image
schemas as families of interrelated logical theories, with each schema comprising a combination
of primitive components, and using QSR, and other, formalisms that capture the spatiotemporal
content of schemas [20]. In our approach, we chose not to use such formalisms to capture the
internal structure of image schemas, and not to predefine specific variants or combinations of
image schemas. Rather, our aim was to allow aspects of the image schemas (for example, the
number of locations on a path, or the number of levels on a scale) to be shaped partly by their
correspondences with the geometry of the diagram [21].
As for diagrammatic reasoning, some literature posits that the effectiveness of diagrams
lies in the fact that certain visual relations in the geometry of the diagram align with certain
information in the domain of reference of the diagram, enabling directly observing additional
information from the geometry, without needing additional inference steps [22, 23]. Here, we
expanded in this direction by modeling the origin of these properties as the blending of image
schemas with the geometry of a diagram.
4. Discussion and Conclusions
The predominant logical approaches to diagrammatic reasoning do not take into account the
user as an embodied being, actively reasoning with the diagram. We thus believe sense-making
with image schemas provides insight into the cognitive aspects of diagrammatic reasoning
and so we set out to model it. We have presented a formal framework of the sense-making of
diagrams, modeling the way users blend their embodied cognitive structures with the geometry
of a diagram, capturing the emergence of inferences [6]. To the best of our knowledge, our
approach is a novel contribution to the literature of both image schemas and diagrammatic
reasoning. Below we describe the directions towards which we are currently extending our
work.
We are interested in using our framework to investigate why some diagrammatic formalisms
are more effective than others for reasoning. In principle, almost any image schema and any
diagram can take part in a network of conceptual blends, with different inferences emerging.
However, some of these inferences may not be valid given the semantics of a particular diagram.
Therefore, a diagram would be effective if there exist some networks of blends of its geometry
and certain image schemas, wherein valid inferences about its semantics can emerge. Another
way to evaluate a blend network, and thus the effectiveness of a diagram, is the formalisation
and use of governing principles for conceptual blending. These principles assess features such as
how tightly integrated a blend is, how much of the structure of the input spaces it incorporates,
and whether the correspondences between elements of the input spaces appear in the blend [5,
ch. 16]. These principles are proposed to show how cognitively useful a blending network is,
and our proposal would be that a diagram π·1 is more effective than diagram π·2 if its geometry
can be blended with image schemas in a blending network that satisfies the governing principles
more than the corresponding blending network for π·2 . This work could provide guidelines for
the design of effective diagrammatic and graphical visualisations by characterising diagrammatic
formalisms as effective in a manner that takes into account our embodiment.
Going in a different, but related, direction, we also plan to implement a pipeline that explores
possible blends. This way we could model the sense-making of diagrammatic and other visual
formalisms, discovering various possible senses of them in a cognitively-inspired way. However,
this may prove to be challenging because the resulting search space is vast [24]. The governing
principles can serve as heuristics to guide our search of blends between image schemas and
diagrams. Our framework can be validated in part by applying it to diagrams used in existing
experimental studies on diagrammatic formalisms and their features, and how they affect
reasoning speed and accuracy. It is possible to check if the blending networks selected by our
pipeline correspond to diagrams that had high effectiveness (high reasoning speed and accuracy)
in behavioral experiments.
Finally, although so far we have modelled a blending process to represent the sense-making of
geometric configurations of diagrams, our model could be applied in other domains. Furthermore,
the entire framework could eventually be generalised in a representation-independent manner
as described in [12]. We believe our work has potential as a general-purpose, parsimonious
module for modeling the way an agent makes sense of its environment, using a few preexisting
structures, i.e., image schemas, together with a search and blending process. Image schemas, or
similar concepts, have indeed been proposed as useful primitives for common-sense reasoning
[25, 26, 27].
References
[1] F. J. Varela, Organism: A meshwork of selfless selves, in: Organism and the Origins of
Self, Springer, 1991, pp. 79β107.
[2] M. Merleau-Ponty, The structure of behavior, Duquesne Univ Press.(Original published in
1942), 1983.
[3] M. Johnson, The Body in the Mind, Univ of Chicago Press, 1987.
[4] G. Lakoff, Women, Fire, and Dangerous Things, Univ of Chicago Press, 1987.
[5] G. Fauconnier, M. Turner, The Way We Think, Basic Books, 2002.
[6] D. Bourou, M. Schorlemmer, E. Plaza, Image schemas and conceptual blending in dia-
grammatic reasoning: the case of Hasse diagrams, in: DIAGRAMS Proc, Springer Nature,
2021.
[7] M. May, Diagrammatic reasoning and levels of schematization, in: Iconicity. A Fundamental
Problem in Semiotics, NSU Press, 1999, pp. 175β194.
[8] E. Astesiano, M. Bidoit, H. Kirchner, B. Krieg-BrΓΌckner, P. D. Mosses, D. Sannella, A. Tar-
lecki, CASL: the common algebraic specification language, Theor Comput Sci 286 (2002)
153β196.
[9] T. Mossakowski, C. Maeder, K. LΓΌttich, The HEterogeneous Tool Set, HETS, in: TACAS
Proc, Springer, 2007, pp. 519β522.
[10] D. Bourou, M. Schorlemmer, E. Plaza, Modelling the sense-making of diagrams using
image schemas, in: CogSci 2021 Proc, UC Merced, 2021.
[11] T. R. Besold, K.-U. KΓΌhnberger, E. Plaza, Towards a computational- and algorithmic-level
account of concept blending using analogies and amalgams, Conn Sci 29 (2017) 387β413.
[12] M. Schorlemmer, E. Plaza, A uniform model of computational conceptual blending, Cogn
Syst Res 65 (2021) 118β137.
[13] T. Froese, T. Ziemke, Enactive artificial intelligence: Investigating the systemic organization
of life and mind, Artif Intell 173 (2009) 466β500.
[14] Z. Falomir, E. Plaza, Towards a model of creative understanding: Deconstructing and
recreating conceptual blends using image schemas and qualitative spatial descriptors, Ann
Math Artif Intell 88 (2019) 457β477.
[15] B. Bergen, N. Chang, Embodied construction grammar in simulation-based language un-
derstanding, in: Construction grammars: Cognitive grounding and theoretical extensions,
volume 3, John Benjamins, 2005, pp. 147β190.
[16] J. E. Bryant, Best-Fit Constructional Analysis, Ph.D. thesis, EECS Department, Univ
of California, Berkeley, 2008. URL: http://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/
EECS-2008-100.html.
[17] M. A. RodrΓguez, M. J. Egenhofer, A comparison of inferences about containers and surfaces
in small-scale and large-scale spaces, J Vis Lang Comput 11 (2000) 639β662.
[18] R. St Amant, C. T. Morrison, Y.-H. Chang, W. Mu, P. R. Cohen, C. Beal, An image schema
language, Technical Report, North Carolina State Univ at Raleigh Dept of Computer
Science, 2006.
[19] W. Kuhn, An image-schematic account of spatial categories, in: COSIT Proc, LNCS,
Springer, 2007.
[20] M. M. Hedblom, Image schemas and concept invention: cognitive, logical, and linguistic
investigations, Springer, 2020.
[21] D. Bourou, M. Schorlemmer, E. Plaza, A cognitively-inspired model for making sense of
Hasse diagrams, in: CCIA Proc, IOS Press, 2021.
[22] A. Shimojima, On the Efficacy of Representation, Ph.D. thesis, Indiana Univ, 1996.
[23] G. Stapleton, M. Jamnik, A. Shimojima, What makes an effective representation of in-
formation: a formal account of observational advantages, J Logic Lang Inf 26 (2017)
143β177.
[24] R. Confalonieri, M. Eppe, M. Schorlemmer, O. Kutz, R. Penaloza, E. Plaza, Upward refine-
ment operators for conceptual blending in the description logic β°β++ , Ann Math Artif
Intell 82 (2018) 69β99.
[25] M. Shanahan, M. Crosby, B. Beyret, L. Cheke, Artificial intelligence and the common sense
of animals, Trends Cogn Sci 24 (2020) 862β872.
[26] M. Thosar, C. A. Mueller, G. Jaeger, M. Pfingsthorn, M. Beetz, S. Zug, T. Mossakowski,
Substitute selection for a missing tool using robot-centric conceptual knowledge of objects,
in: SAC Proc, ACM, 2020, pp. 972β979.
[27] M. M. Hedblom, M. Pomarlan, R. Porzel, R. Malaka, M. Beetz, Dynamic action selection
using image schema-based reasoning for robots, in: JOWO Proc, 2021.