Visualising Image Schemas: A Preliminary Look at the Diagrammatic Image Schema Language (DISL) Maria M. Hedblom1 , Fabian Neuhaus2 1 Jönköping Artificial Intelligence Laboratory, Jönköping University, Sweden 2 Research Group Theoretical Computer Science, Otto-von-Guericke University Magdeburg, Germany Abstract In this extended abstract, we present the Diagrammatic Image Schema Language (DISL). DISL is a structured visual language to represent image-schematic relationships in different situations and lin- guistic expressions. Building on previous visualisation research, as well as how image schemas can be ontologically organised, DISL offers a more systematic representation language for abstract concepts than previously possible. In particular, we discuss some of the underlying motivations and design decisions for DISL and introduce the conceptual primitives that are covered by DISL. Their use is illustrated with the classic example of Object_Into_Container. Keywords image schemas, theory graphs, conceptual primitives, conceptual visualisation 1. Introduction Image schemas are mental patterns learned from perceptual experiences that map embodied, spatiotemporal relationships onto conceptual constructions [1]. Despite their name, image schemas are not images, but cognitive gestalts whose cognitive structure can be visualised through mental imagery and, thus, can be illustrated diagrammatically. They capture the conceptual information of relationships between objects and their environments with commonly mentioned examples being concepts such as Containment, Support and Source_Path_Goal. In cognitive linguistics, the underlying information in image schemas is occasionally explained with the help of illustrations, which show the salient features of a given image schema. Notable mentions are the work of Johnson’s introductory illustrations [1], the image-schematic drawings by Mandler [2], Hurtienne et al. [3] and Talmy’s visual representations of force-dynamics [4], and the rather comprehensive visual grammar used to describe linguistic relationships by Langacker [5]. Work like these, demonstrate the desire for a visual language to illustrate the semantic content of the image schemas. However, it also showcases the complexity of such endeavours as different representation methods more often than not are visually disjoint. Additionally, these representations of image schemas in the literature are typically ad hoc in the sense that there appear to be no general principles that guide these illustrations. The Sixth Image Schema Day (ISD6), March 24–25, 2022, Jönköping University, Sweden Envelope-Open maria.hedblom@ju.se (M. M. Hedblom); fneuhaus@ovgu.de (F. Neuhaus) Orcid 00070-0001-8308-8906 (M. M. Hedblom); 0000-0002-1058-3102 (F. Neuhaus) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) To address this issue, we develop the Diagrammatic Image Schema Language (DISL). Used for the diagrammatic representation of image schemas, it focuses on systematising the visual combinations rules. DISL takes inspiration from previous visualisations of image schemas and combines it with the research on how image schemas can be divided into compositional primitives [6] and the research that showcase how based on the primitives the image schemas can be structured into hierarchical family graphs [7, 8]. Together this foundation offers a more systematic and compositional methodology to represent both individual image schemas as well as the events that take place with them. In order to represent the image schemas, DISL defines a set of symbols for the representation of some conceptual primitives, the recurring building blocks of image schemas, and further defines principles how to compose an image schema from these components into spatial stories. This means that DISL can be used to analyse complex events and diagrammatically show how different image schemas interact with one another in real-life scenarios. We believe this to be beneficial for many areas, not the least in research on cognitive interpretations of visual image- schematic metaphors (e.g [9]), research on spatial expressions and configurations (e.g. [10]) and research in cognitive robotics on events and action segmentation based on image-schematic reasoning (e.g. [11, 12]). The purpose of this paper is to provide a preliminary view of the approach we have taken, show a selection of conceptual primitives that we have chosen, show their diagrammatic representation, and illustrate their use with an example. 2. Image Schemas and Their Compositions Human conceptualisations rely partially on recurring and embodied patterns of spatiotempo- ral relationships, which are often referred to as image schemas [1]. Mandler defines image schemas as “...dynamic analog representations of spatial relations and movements in space” [2, p. 591]. Further the conceptual structure within the image schemas can be divided into smaller components. In [13], Mandler and Cánovas introduce the following distinction: Spatial primitives are the first building blocks that allow us to understand what we perceive, e.g., Container, Contact, Open. Image schemas are representations of simple spatial events using the primitives, e.g., Ob- ject_Into_Container. Schematic integrations are conceptual representations to include non-spatial elements, by projecting feelings or non-spatial perceptions to blends structured by image schemas, e.g., Blocked_Motion. This distinction allows to represent image schemas as temporal stories with different ‘parts.’ However, many traditional image schemas do not represent temporal stories in that they contain an (obvious) change over time. Support is one such example, that while a temporal dimension is needed to determine the physical stability of the supporting object, it is still possible to identify Support in an instance. To account for these ‘static manifestations of image schemas’ we extend the classification in [13] as follows: Primitive amalgamation The integration of several conceptual primitives into a whole rep- resenting a static combination of entities, their relationships and attributes, e.g., Support. The difference between image schemas and conceptual primitives is of particular interest to the work in this paper, as it relies on how spatial primitives provide the elements that are used to create the DISL diagrams. However, as the three examples for spatial primitives mentioned above (namely, Container, Contact, Open) illustrate, conceptual primitives are a divers group of structures, which play different roles within image schemas. If we consider image schemas as kind of ‘spatial stories’, then some conceptual primitives play the role of the ‘characters’, or predicates in linguistics, (e.g., an Object and a Container), some conceptual primitives describe the relationships between the ‘characters’ (e.g., the Object is Contained inside the Container), and some attributes of the ‘characters’ (e.g., the Container is Open). The image schema unfolds as ‘characters’ enter or leave the stage, and their relationships and attributes change over time. Since the different roles of the conceptual primitives determine their roles in the diagrammatic representation, we suggest to further classify them as follows: Independent primitives may occur in an image schema independently of other primitives, e.g., Object, Container, Path, Spatial Region. Relational primitives define the relationship between two (or more) independent primitives, e.g., Contact, Link, Contained. They are dependent on their relata in the sense that they cannot be present without the independent primitives they relate. Attributive primitives define attributes of another conceptual primitive, e.g., Open/Closed are attributes of a Container’s openings and Moving and At_Rest are attributes of Objects. Independent, relational and attributive primitives provide the basic building blocks for com- posing primitive amalgamations, image schemas and schematic integrations. 3. Visualising the Primitives Our diagrammatic representation of image schemas and schematic integrations follows inspira- tion from work [14, 15] that utilise the narrative conventions of comic strips. Comic strips utilise a sequence of panels to tell a narrative in a condensed form. An individual panel represents either a moment in time or some extended time period without significant qualitative change. In DISL, an individual panel either visualises a conceptual primitive or a schematic integration of several conceptual primitives. A sequence of these panels is utilised to represent dynamic change and, thus, represent image schemas or schematic integrations. Inspired by Langacker’s boxed representations of the transition of image schema states [5], we use similar stylistic elements to characterise image-schematic narratives. In this section, we present an overview of the conceptual primitives that we currently selected for DISL and their symbolic representations. 3.1. Independent primitives Independent conceptual primitives are the cognitive primitives that can be added to a panel without requiring the existence of another primitive. Figure 1 contains an overview of the symbolic representations of these independent cognitive primitives. Following convention (e.g. [1]), we represent objects as shapes with solid matter by adding colours and borders, the Path is a line, following Johnson’s and Mandler’s depictions [1, 2]. Figure 1: Graphical representation of some independent conceptual primitives. 3.2. Relational primitives Relational conceptual primitives express the relationship between two independent conceptual primitives. A basic spatial primitive is Location, which is the relationship between some object and the region of space it occupies within the context of a scene (including other objects). Location is represented within a panel by the way the object is located within the frame of the panel in the context of other independent conceptual primitives (e.g., objects and paths). In Figure 2, some relational primitives are depicted. Figure 2: Graphical representation of some relational primitives. 3.3. Attributive primitives In Figure 3, the attributive primitives are depicted as ‘crowns’ for independent primitives. The focus lies on discussing attributes that are related to either Motion, the embodied experience of force acting on the body, the force-dynamic primitive Umph, or attributes of Containers. Motion is the attributional conceptual primitive of motion of an object in some directed form. (Undirected motion, e.g., a trembling animal, would not instantiate Motion.) Motion is represented by an arrow, which indicates the trajectory. Figure 3: Graphical representation of some attributional primitives focusing on movement of objects. 4. Illustration of DISL with an example Figure 4 illustrates how in DISL the representation of conceptual primitives are combined to represent image schemas diagrammatically. Individual scenes are represented by panels, which represent amalgamations of several conceptual primitives. Following Western conventions, the order of the scenes, from left to right, represent temporal succession. The first scene contains one Object, one Container and one Path with an end (an attributive primitive). The Container has an open border. The spatial relations between the primitives are represented by their representation in the scene, in particular: the Object and the Container are disconnected, the Object is on the Path, and the end of the Path is contained within the Container. Further, the Object has the image schema Motion_Along_Path, in the direction of the end of the path. Thus, the first panel contains symbols for three independent primitives and three attributional primitives. The next two panels represent other amalgamations that differ from the previous one by significant qualitative changes of attributes of the independent primitives or their spatial relations. In the second panel, the Object crosses the boundary of the Container. In the third panel, the object is no longer moving on the Path but is at rest contained within the Container. Together, the whole diagram represents the image schema Object_Into_Container. Note that Object_Into_Container leave much information unspecified. E.g., it is possible that the Container is in motion or at rest, that the Object is moving on its own or that it is caused to move, that the Container is occupied or that it is full at the end of the process. By adding additional information in the form of additional primitives, one can refine the information present in the image schema. For instance, one could refine Object_Into_Container by Figure 4: DISL representation of Object_Into_Container specifying the movement of the Container as Caused_Motion in panels 1 and 2 and specifying the container as Full in the third panel. This would yield an image-schematic representation of something like Container_is_Filled. 5. Discussion and Future Work This extended abstract introduced some of the visual components of the Diagrammatic Image Schemas Language, DISL. We illustrated its use with a help of an example. In the full paper on DISL, we are going to provide rules that guide the combination of conceptual primitives into primitive amalgamations, image schemas and schematic integrations. Further, we will discuss the intended semantics of the symbolic representations of the primitives, and extend our approach to include some important force primitives. Another important aspect that we did not address in this abstract is diagrammatic inferences with DISL. Namely, how it follows in Figure 4 that the Container in the third panel is not empty. This too will be addressed in more extended work. References [1] M. Johnson, The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason, The University of Chicago Press, Chicago and London, 1987. [2] J. M. Mandler, How to build a baby: II. Conceptual primitives, Psychological review 99 (1992) 587–604. [3] J. Hurtienne, D. Löffler, P. Gadegast, S. Hußlein, Comparing pictorial and tangible notations of force image schemas, in: Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, 2015, pp. 249–256. [4] L. Talmy, Force dynamics in language and cognition, Cognitive science 12 (1988) 49–100. [5] R. W. Langacker, Foundations of cognitive grammar: Theoretical prerequisites, volume 1, Stanford university press, 1987. [6] J. M. Mandler, C. P. Cánovas, On defining image schemas, Language and Cognition (2014) 1–23. [7] M. M. Hedblom, O. Kutz, F. Neuhaus, Choosing the Right Path: Image Schema Theory as a Foundation for Concept Invention, JAGI 6 (2015) 22–54. [8] M. M. Hedblom, Image Schemas and Concept Invention: Cognitive, Logical, and Linguistic Investigations, Cognitive Technologies, Springer Computer Science, 2020. [9] J. M. Cunha, P. Martins, P. Machado, Using image schemas in the visual representation of concepts., in: TriCoLore (C3GI/ISD/SCORE), 2018. [10] U.-J. Rüetschi, S. Timpf, Using image schemata to represent meaningful spatial configu- rations, in: OTM Confederated International Conferences” On the Move to Meaningful Internet Systems”, Springer, 2005, pp. 1047–1055. [11] K. Dhanabalachandran, V. Hassouna, M. M. Hedblom, M. Küempel, N. Leusmann, M. Beetz, Cutting events: Towards autonomous plan adaption by robotic agents through image- schematic event segmentation, in: Proceedings of the 11th on Knowledge Capture Confer- ence, 2021, pp. 25–32. [12] M. Pomarlan, J. A. Bateman, Embodied functional relations: A formal account combining abstract logical theory with grounding in simulation, in: Formal Ontology in Information Systems: Proceedings of the 11th International Conference (FOIS 2020), volume 330, IOS Press, 2020, p. 155. [13] J. M. Mandler, C. P. Cánovas, On defining image schemas, Language and Cognition 6 (2014) 510–532. [14] M. M. Hedblom, O. Kutz, R. Peñaloza, G. Guizzardi, Image schema combinations and complex events, KI-Künstliche Intelligenz 33 (2019) 279–291. [15] T. Akimoto, Cogmic space for narrative-based world representation, Cognitive Systems Research 65 (2021) 167–183.