information in notions such as Containment, Source_Path_Goal and Scale. The theory originated from cognitive linguistics as a means to explain the vast prevalence of embodied metaphors and spatial language. However, it has become a valid contribution in many areas investigating the nature of thought. For formal analysis of image schemas, the abstract and undetermined notions in the image schemas require precise and concrete representations. To deal with this, the image schemas can be decomposed into diferent types of conceptual primitives. By adding or removing these primitives, the schematic narrative changes in subtle but essential ways. To demonstrate the power of using a methodology that isolates these primitives, this paper presents a formal analysis of the transfer of forces and motion by looking at the semantic diferences in a selected number of synonyms for 'pushing'. The analysis is done using the visualisation tool The Diagrammatic Image Schema Language and the notions are formalised
Embodied cognition proposes that conceptual grounding takes place through the body’s
sensorimotor interactions with its environment [
To be able to explain how this abstraction phenomenon still remains relevant for more complex conceptualisation, the theory of image schemas ofers a good foundation [ 3, 4]. Image schemas are the abstract generalisations of experience in the format of spatiotemporal relationships between objects, agents and environment. For instance, the conceptual space of ‘car’ is a conceptual combination of its prototypical visual and physical aspects (such as having four wheels and an engine) but the most relevant feature of a car is the functionality based on the spatiotemporal relationships Containment and Source_Path_Goal. These are the imageschematic relationships that ground the concept into its functionality: first, being able to contain people and, second, being able to transport them from place a to b.
These kinds of patterns are proposed to underlie more or less all forms of thought in the form of embodied metaphors [5]. Take, for example, a situation like going to jail where the underlying pattern can be described as a ‘movement into a closed container’. Naturally, a lot of information and associations to these concepts are also important for the full conceptualisation but to understand certain conceptual diferences, the image schemas provide the skeleton of the most vital information.
These kinds of patterns were originally introduced in cognitive linguistics to motivate the number of spatial and embodied metaphors but have since then become an interest of study in a range of diferent scientific disciplines (e.g. developmental psychology [ 6], interaction design [7], gesture analysis [8] and cognitive robotics [9]).
Due to their ability to act as a cognitive bridge between mental conceptualisations and real experiences in the world, there has been an increased interest in utilising these patterns in intelligent systems research. For this to be possible, the abstract generalisation of the image schemas needs to be treated in a formal and systematic manner in order for the representations to be machine-readable and usable in computational systems.
In previous work, briefly described below, we have been working on concretising the abstract nature of the image schemas by building on the idea that individual image schemas can be dissected into the compositional involvement of conceptual primitives.
To demonstrate the power of using a more formal specification level when doing research on image schemas, this paper presents an image-schematic analysis based on conceptual primitives involved in the action concept Push by comparing the semantics of a few selected synonyms. The formal representations will be conducted using The Diagrammatic Image Schema Language (DISL) [10] and The Image Schema Logic (ISL ) [11]. Our main argument is that a formal method of this level of specificity allows the concepts to be presented at a very high level of granularity and, thus, can capture the minuscule diferences in semantic comprehension.
Learned from embodied experiences from early infancy, image schemas are thought to be the conceptual building blocks that structure our understanding [6]. In the form of schematic gestalts, they capture the structural contours of sensorimotor experiences [12] and lay the foundation of our analogical and metaphoric reasoning [13]. As an example, consider the abstract transfer of information in how a text contains information, even though there is no obvious physical Container.
Due to this gestalt structure, it is – from a cognitive perspective – impossible to remove components from the image-schematic concept [14]. As an example, consider a Container without either an insider or an outside. While this may be true for the cognitive domain, primitives such as insides and outsides are very useful in isolation when describing increasingly ifne-tuned situations and concepts.
Within cognitive linguistics, image schemas are studied to core down the semantic content of concepts. For instance, [15] claimed that for any understanding of the word ‘put’, the understanding of the image schema Containment is required. However, there are also several types of both static [16] and dynamic containment relationships [17]. In truth, substantial work has been conducted in this area, for instance, to analyse action verbs (e.g. ‘lifting’ [ 18], ‘standing/lying’ [19]), prepositions (e.g. ‘over’ [20]) and spatial relationships (e.g. ‘straight ’ [21]) that produce semantically distinct representations of what the diferent words contain. However, empirical work has also looked into the human conceptualisation of the semantic prevalence of these kinds of patterns. For instance, in the experiment in [22], it was shown that the Source_Path_Goal schema was conceptually decomposed and that goals and sources could conceptually be ‘removed’ from the movement schema. A phenomenon that was also seen in the corpus study in [23] on abstract movement manifestations.
When multiple combinations of image schemas are required for a particular concept or narrative to be properly understood, this is referred to as image schema profiles [ 24]. From a formal perspective, these combinations come in diferent forms (for an outline see [ 25]), but for the representations of temporal scenes or narratives, it is enough to speak of structured image schema profiles ; ie. combinations that are sequentially ordered.
Highlighted in this section is how the subtle diferences that materialise due to the conceptual primitives can completely change the concept’s meaning. Such diferences in semantic finetuning of the concepts are of great importance to second language learners (see [26]) or, as the main focus in this study is, to better approach computational systems.
Primarily a topic in linguistics, research devoted to the formalisation and computational application of image-schematic concepts is becoming increasingly common. Formal accounts include work on using the spatial information in geographical information science [27], modelling agent behaviour and functional relationships [28, 29], their role in metaphors [30] but also pure formalisations of the image schemas can be found [16, 31].
Despite being atomic concepts from a cognitive perspective, from a logical perspective, the image schemas can be decomposed into smaller components. Returning to the notion of Containment, from a real-world perspective, it is impossible to speak about an outside without also considering an inside and a border between these regions. However, this compositionality can be omitted to favour a more open-ended interpretation as demonstrated with the Source_Path_ Goal members in which goals and sources could be ‘conceptually removed’ [22].
Using such empirical results as an inspiration, we proposed that what is traditionally seen as ‘one’ image schema can be decomposed into a family of closely related notions that are structured hierarchically based on the addition or removal of conceptual primitives [32]. This WORD
Push Nudge Touch
Tap
Poke Punch Press Shove
Relational primitives contact, contact, ¬contact contact, contact, up, ¬contact, contact, contained contact, contained, ¬contact contact, up, ¬contact, scale contact, up, ¬contact
Attributive primitives Umph, Self+ Caused_Motion Self_Motion Umph Umph, Self_Motion Umph, Self_Motion Umph, Self+ Caused_Motion Umph, Self_Motion Umph, Self+ Caused_Motion
Interpretation continuous pushing soft touch continuous touch touch from above soft push into push into with move above push that shrinks push up with move structure enables the abstract schema ‘prototype space’ of individual image schemas to be formally approached by the simple composition of conceptual primitives.
Based on this we introduced The Image Schema Logic (ISL ), a combination logic that can represent (many of) the image-schematic primitives and, in so doing, be combined to represent structure image schema profiles [ 33, 11]. ISL is a combination logic that can represent several types of spatiotemporal relationships between agents, objects and environments. It builds on Region Connection Calculus (RCC8) [34] to represent the spatial relationships, Qualitative Trajectory Calculus (QTC) [35] to describe the relative movement, Linear Temporal Logic over the Reals (ℝ ) to capture the temporal dimension and uses First-Order Logic (FOL) as a concept language.
Based on the same compositional principle, we also introduced The Diagrammatic Image Schema Language (DISL) [10]. DISL is a visual representation language that can be used as a complement to the study of image schemas in which conceptual primitives are combined to represent the image-schematic skeleton of events. Based on commonly studied notions, DISL consists of independent, relational and attributive primitive types that are further categorised based on their nature into spatial, spatiotemporal and force dynamic. To represent the structure image schema profiles, each of these primitive types is subject to diferent types of combination rules that enable the representation of the intended semantic content over the time axis.
Image schemas are also useful for understanding the semantic structure of events. By decomposing the semantics of events into their respective image schema transitions, we introduced image-schematic event segmentation [36]. In that work, the main hypothesis is that ‘cognitive’ event segmentation happens in the change of image-schematic state, and not due to particular time periods or similar [37]. By looking at events in this manner, the formal decomposition of events into image-schematic scenes becomes rather straightforward. To explain this, we will combine these representation methods and look at a few action concepts involving the transfer of force and movement from an active agent to a passive object/agent as seen in ‘pushing’.
Pushing is (here) defined as the transfer of force from one active agent onto a passive object/agent causing it to continuously move. We reserve ourselves to the shortcoming that we have not conducted any empirical study as to whether this is the most ‘correct’ way to interpret pushing and the chosen pushing synonyms. Instead, we ask the reader to contemplate the methodologies by which it is possible to represent minuscule diferences in language and conceptualisation and not focus on the details of that conceptualisation. In Table 1, the synonyms are presented as well as the primitives that are involved throughout the action event.
To visually represent these minor diferences, DISL systematically distinguish between independent conceptual primitives (e.g. object, container, path), relational conceptual primitives (e.g. contact, bigger_than) and attributive conceptual primitives (e.g. types of movement, and passive and active (p-/a-)Umph1). DISL takes the open-world assumption, meaning that what is not explicitly expressed is either unknown or not relevant. In Figure 1, the relevant primitives for the modelled narratives are presented (for a more comprehensive list see [10, 39]). Worth noting is that objects can take any representative shape, the ‘ground’ symbol denotes Vertical orientation, not per se Contact between the ground and the objects. Further, force, Umph, is distinguished as the active (internal) force of an active agent with arrows pointing from the centre (a-Umph) or as the passive (external) force onto a passive object/agent with arrows pointing inwards from the border (p-Umph).
Table 1 lists in a simplified manner the diferent conceptual primitives that take place in the action concept. Already on this level, it becomes quite clear that each action event is (here) conceptualised using diferent combinations of primitives. However, since these are events, the presence or absence of the primitives needs to be represented on a time axis. DISL does this by using a strip consisting of temporally ordered (from left to right) panels that each represent the image-schematic scene of the ongoing event. A panel can be instantaneous or infinitely long. The change in the panel is entirely due to the change in the image-schematic relationships. 1Umph is the experience of force [38]. We here distinguish between active/internal and passive/external force.
To demonstrate this visual language, Figure 2 presents the action concepts once more, this time visualised using DISL. The state at time 0 describes the required starting point for the event to take place but is not part of the action event itself. In most cases, the relationship between the two objects remains the same in all pushing synonyms with one object moving towards the other. Shove is the one exception, included to demonstrate that for active agents it is also possible for movement transfer to be conducted through force that manifests without (obvious) movement from the active agent. The action verbs also consist of diferent amounts of image-schematic segments. In the representation, Press requires four panels, whereas Push (in this interpretation) only require one panel as the concept is interpreted as continuous. To which degree the final scene of each action concept is ‘part of’ the event is also worth considering in more detail, but for now, the end results are considered to be a pivotal component of the action itself. Note that darker colours highlight significance, here ‘the active agent’ or the intensity of the force, as inspired by previous visualisation techniques of language [40].
Further, relational primitives such as Contact and relative size as in the image schema Scale, are represented using the independent objects’ positioning and size within the DISL panels. For instance, the ‘pressed object’ can be seen to shrink in the process by becoming smaller than the ‘pressing object’.
In order to turn DISL or any image schema representation, into a machine-readable format, a language like ISL ofers a bridge. Therefore, using ISL annotations, we have formalised the eight mentioned push synonyms in the formalisations below.
In ISL , variables can be either Objects or Regions. As we are exclusively dealing with Objects in this article, ∀ 1, 2∶ has been removed from all formalisations. Another important symbol is U, which is a temporal sequential operator to represent “until”. Semantically, it corresponds to the change of panel in DISL. Cognitively, it describes that the previous imageschematic state continues ‘until’ there is a new state (as in the ceasing of movement due to blockage). The symbols ⇝ and ↩ represent the first object (x) moving to and from the second object (y). It might feel unintuitive that objects can move towards one another even while in Contact. This is due to the semantics ISL in which an object ‘exists’ at its centre, for documentation of the complete semantics we refer to [11]. To describe force relationships, when a-Umph(x) → p-Umph(y), interpreted as logical implication (essentially when a-Umph(x) leads to p-Umph(y)), we represent this by using the joint term forces(, ).
Note also that the pre-scene 0 has not been included in the ISL formalisation as it mostly captures the same relationship and was argued to not play a role in the action concept.
Push( 1, 2) ↔ (In_Contact( 1, 2) ∧ forces( 1, 2) ∧ Caused_Motion( 2) Nudge( 1, 2) ↔ In_Contact( 1, 2)
U ( 1 ↩ 2 ∧ ¬In_Contact( 1, 2)) Touch( 1, 2) ↔ In_Contact( 1, 2) ∧ forces( 1, 2)
U (In_Contact( 1, 2) ∧ ¬forces( 1, 2)) Tap( 1, 2) ↔ above( 1, 2) ∧ In_Contact( 1, 2) ∧ forces( 1, 2)
Poke( 1, 2) ↔ In_Contact( 1, 2) ∧ ( 1, 2) ∧ forces( 1, 2) Punch( 1, 2) ↔ In_Contact( 1, 2) ∧ ( 1, 2) ∧ forces( 1, 2)
U ( 1 ↩ 2 ∧ ¬In_Contact( 1, 2))
U(inside( 1, 2) ¬( 1, 2) U ( 1 ↩ 2 ∧ ¬In_Contact( 1, 2))) U (inside( 1, 2) ¬( 1, 2) ∧ 2 ↩ 1
U (¬In_Contact( 1, 2) ∧ 1 ↩ 2)) Press( 1, 2) ↔ above( 1, 2) ∧ 1 ⇝ 2 ∧ In_Contact( 1, 2) ∧ forces( 1, 2)
U (In_Contact( 1, 2) ∧ forces( 1, 2) ∧ shrink( 1) Punch( 1, 2) ↔ In_Contact( 1, 2) ∧ 1 ⇝ 2 ∧ forces( 1, 2)
U ( 1 ↩ 2
U (¬In_Contact( 1, 2)))) U( 2 ↩ 1 ∧ above( 1, 2)
U(¬In_Contact( 1, 2)))
This paper formally modelled action concepts related to ‘pushing’ by using DISL and ISL as formal representation languages. The main purpose of this is to demonstrate the usefulness of looking at image schema narratives as sequential compositions of conceptual primitives. Through subtle changes in the involved conceptual primitives, we showed how the semantics of the narratives change rather fundamentally. This means that even though image schemas are considered gestalts [12, 3], they are required to be interpreted as compositions of conceptual primitives [41, 32].
That said, the modelling work in this paper is not without faults. Compared to the work in [16] respective [23] where the authors use linguistic corpora to extract synonyms of Containment respectively Source_Path_Goal, the interpretation of the action concepts in this paper is ad hoc. This reduces the usability of the presented formalisms, but the paper’s contribution lies in the value of using compositional conceptual primitives to express improved semantic granularity.
The formalisation is based on two diferent languages, DISL and ISL . ISL was introduced as a combination logic to capture (some of) the spatiotemporal aspects of the image schemas in regard to their primitives. However, the abstract notions of the image schemas and the (near impossible) complexity of formally modelling dynamic and transformational aspects of these notions, both the syntax and the semantics of ISL fall short on many accounts. Dealing with this, a large portion of ISL ’s concept language is thus written in first-order logic. While this is highly expressive, and essentially any relationship can be modelled, it is also devoid of the required semantic grounding for more applicable machine-readability.
In [42], we tried to translate the information in ISL to the description logic EL++ in order for the structural image schema profiles to be used as semantically grounded action descriptors in cognitive robotics. While possible, the expressiveness in ISL was lost in the process.
In future work, we hope to better capture the expressibility of ISL into a machine-readable format in order to ofer a conceptual bridge between real-world examples and the semantics of experience. and concepts: Theoretical views and inductive data analysis, Cognitive science series, Academic Press, 1993, pp. 67––95. [3] M. Johnson, The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason,
University of Chicago Press, Chicago, US, 1987. [4] G. Lakof, Women, Fire, and Dangerous Things. What Categories Reveal about the Mind,
University of Chicago Press, Chicago, US, 1987. [5] G. Lakof, M. Johnson, Metaphors We Live By, University of Chicago Press, 1980. [6] J. M. Mandler, The Foundations of Mind: Origins of Conceptual Thought: Origins of
Conceptual Though, Oxford University Press, New York, 2004. [7] J. Hurtienne, J. H. Israel, Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction, in: Proceedings of the 1st international conference on Tangible and embedded interaction, ACM, 2007, pp. 127–134. [8] I. Mittelberg, Enacted schematicity: Image schemas and force dynamics operating in gestural (inter-) action., in: Proc. of TriCoLore, volume 2347 of CEUR-WS, Bolzano, Italy, 2019. [9] Y. Nagai, M. Asada, Predictive learning of sensorimotor information as a key for cognitive development, in: Proc. of the IROS 2015 Workshop on Sensorimotor Contingencies for Robotics, volume 25, 2015. [10] M. M. Hedblom, F. Neuhaus, T. Mossakowski, The diagrammatic image schema language,
Spatial Cognition and Computation (2023). Forthcoming. [11] M. M. Hedblom, Image Schemas and Concept Invention: Cognitive, Logical, and Linguistic
Investigations, Cognitive Technologies, Springer Computer Science, 2020. [12] B. Hampe, Image schemas in cognitive linguistics: Introduction, in: B. Hampe, J. E. Grady (Eds.), From perception to meaning: Image schemas in cognitive linguistics, Walter de Gruyter, 2005, pp. 1–14. [13] Z. Kövecses, Metaphor: A Practical Introduction, Oxford University Press, Oxford, USA, 2010. [14] G. Lakof, R. Núñez, Where Mathematics Comes From: How the Embodied Mind Brings
Mathematics into Being, Basic Books, New York, US, 2000. [15] P. Pauwels, By Word of Mouth: Metaphor, metonymy and linguistic action in a cognitive perspective, John Benjamins Publishing Company, Amsterdam, 1995, pp. 125–158. [16] B. Bennett, C. Cialone, Corpus Guided Sense Cluster Analysis: a methodology for ontology development (with examples from the spatial domain), in: P. Garbacz, O. Kutz (Eds.), 8th Int. Conf. on Formal Ontology in Inform. Systems, volume 267, IOS Press, 2014, pp. 213–226. [17] R. Dewell, Dynamic patterns of containment, From perception to meaning: Image schemas in cognitive linguistics (2005) 369–393. [18] P. Vernillo, The role of the image schemas in the analysis of the semantic variation of action verbs. data from imagact., in: TriCoLore (C3GI/ISD/SCORE), 2018. [19] C. Serra-Borneto, Cognitive linguistics in the Redwoods: The expansion of a new paradigm in linguistics, Cognitive Linguistics Research, Mouton de Gruyter, Berlin, 1996, pp. 459–505. [20] C. Brugman, G. Lakof, Cognitive topology and lexical networks, in: Lexical ambiguity resolution, Elsevier, 1988, pp. 477–508. [21] A. Cienki, STRAIGHT : An Image Schema and its metaphorical extensions (1998) 107–149. [22] L. Lakusta, B. Landau, Starting at the end: The importance of goals in spatial language,
Cognition 96 (2005) 1–33. [23] D. Gromann, M. M. Hedblom, Breaking Down Finance: A method for concept simplification by identifying movement structures from the image schema Path-following, in: Proc. of the 2nd Joint Ontology Workshops, Annecy, France, 2016. [24] T. Oakley, Image schema, in: D. Geeraerts, H. Cuyckens (Eds.), The Oxford Handbook of
Cognitive Linguistics, Oxford University Press, Oxford, 2010, pp. 214–235. [25] M. M. Hedblom, O. Kutz, R. Peñaloza, G. Guizzardi, Image schema combinations and complex events, Künstliche Intelligenz Special issue on Cognitive Reasoning (2019) 279–291. [26] J. Yu, A brief review of the efects of image schema theory on the acquisition of polysemy,
Journal of Language Teaching 2 (2022) 1–6. [27] W. Kuhn, An Image-Schematic Account of Spatial Categories, in: S. Winter, M. Duckham, L. Kulik, B. Kuipers (Eds.), Spatial Information Theory, volume 4736 of LNCS, Springer, 2007, pp. 152–168. [28] R. St. Amant, C. T. Morrison, Y.-H. Chang, P. R. Cohen, C. Beal, An image schema language, in: Int. Conf. on Cognitive Modeling (ICCM), 2006, pp. 292–297. [29] M. Pomarlan, J. Bateman, Embodied functional relations: a formal account combining abstract logical theory with grounding in simulation, in: 11th Int. Conf. on Formal Ontology in Information Systems, 2020. [30] Y. Kaneko, K. Okada, S. Ito, T. Nomura, T. Takagi, A proposal of analogical reasoning based on structural mapping and image schemas, in: SCIS & ISIS SCIS & ISIS 2010, Japan Society for Fuzzy Theory and Intelligent Informatics, 2010, pp. 328–332. [31] A. U. Frank, M. Raubal, Formal specification of image schemata – a step towards interoperability in geographic information systems, Spatial Cognition and Computation 1 (1999) 67–101. [32] M. M. Hedblom, O. Kutz, F. Neuhaus, Choosing the Right Path: Image Schema Theory as a Foundation for Concept Invention, Journal of Artificial General Intelligence 6 (2015) 22–54. [33] M. M. Hedblom, O. Kutz, T. Mossakowski, F. Neuhaus, Between contact and support: Introducing a logic for image schemas and directed movement, in: 16th Int. Conf. of the Italian Association for Artificial Intelligence (AI*IA 2017), 2017, pp. 256–268. [34] D. A. Randell, Z. Cui, A. G. Cohn, A spatial logic based on regions and connection, in: Proc. of the 3rd Int. Conf. on the Principles of Knowledge Representation and Reasoning (KR’92), Morgan Kaufmann, Los Altos, 1992, pp. 165–176. [35] N. Van De Weghe, A. G. Cohn, G. De Tré, P. D. Maeyer, A qualitative trajectory calculus as a basis for representing moving objects in geographical information systems, Control and cybernetics 35 (2006) 97–119. [36] M. M. Hedblom, O. Kutz, R. Peñaloza, G. Guizzardi, Image schema combinations and complex events, KI-Künstliche Intelligenz 33 (2019) 279–291. [37] J. M. Zacks, K. M. Swallow, Event segmentation, Current directions in psychological science 16 (2007) 80–84. [38] J. M. Mandler, The spatial foundations of the conceptual system, Language and Cognition 2 (2010) 21–44. [39] M. M. Hedblom, F. Neuhaus, Visualising Image Schemas: A Preliminary Look at the Diagrammatic Image Schema Language (DISL), in: Sixth Image Schema Day 2022, Jönköping, Sweden, March 24-25th, 2022, CEUR-WS, 2022. [40] R. W. Langacker, Foundations of cognitive grammar: Theoretical prerequisites, volume 1,
Stanford University Press, 1987. [41] J. M. Mandler, C. P. Cánovas, On defining image schemas, Language and Cognition (2014) 1–23. [42] M. M. Hedblom, M. Pomarlan, R. Porzel, R. Malaka, M. Beetz, Dynamic action selection using image schema-based reasoning for robots, in: Proceedings of the 7th Joint Ontology Workshops, 2021.