This short paper continues work on primitive decomposition systems for meaning representation which combine image schemas and conceptual dependency primitive systems. An important thread of this research seeks small abstract sets of conceptual primitives so that decompositions of imagery evoked by language give rise to rich sets of mappings between language and the language-free representations, reflecting the linguistic variation of human language behavior. In this brief paper, we present a proposal for novel primitive decompositions of positions, spatial relationships, and orientations of objects in space in a conceptual representation framework. As an abstract first approximation, we introduce a spatial primitive which represents that one object is positioned in between two other objects, and combine it with part-whole representations to decompose commonly referenced concepts and language expressions of the positions and orientations of objects in relation to their surroundings.
Natural language understanding is a human intelligence task that continues to be a critical area
for evaluating knowledge representation systems, commonsense reasoning, meaning
representation systems, and ontologies. This short paper continues work on primitive decomposition
systems for meaning representation which combine image schemas [
Much of the focus of earlier work on conceptual dependency was on the representations of events and was dominated by decompositions of eleven primitive physical, mental, and social “acts” which changed the state of the world: PTRANS, MOVE, PROPEL, INGEST, EXPEL, GRASP, ATRANS, MTRANS, ATTEND, MBUILD, and SPEAK. This was largely due to the early focus of applying the representation system towards natural language understanding and story understanding tasks, for which representing dynamic and pivotal story events is key [6].
One relatively unexplored idea is to subject aspects of “static” scenes, spatial characteristics of objects, the spatial relationships between objects, and arrangements of objects in scenes to primitive decompositions. In the spirit of earlier work on CD and more recent work juxtaposing CD primitives with image schemas [7], we are seeking a small abstract set of conceptual primitives and decompositions that will have the same positive benefits as the primitive decompositions for events: it will allow for imagery evoked by language to be represented as unambiguously as possible through a language-free representation, allow rich sets of mappings between language and the representations, and allow for reasoning about scenes at both low levels and high levels of detail, depending on the “molecular” complexity of the primitive-decomposed structures.
We build on prior work on the Mental Motion Pictures system [8], a novel CD-based conceptual analyzer of natural language which attempts an in-depth understanding of ProPara [9] paragraphs by creating sequences of frames to represent the evolution in time of the imagery evoked by the text. Here we focus specifically on how to represent positions, spatial relationships, and orientations of objects in the scenes, whether they are still or in motion.
The conception of physical objects in the system builds on conceptual dependency (CD), image schemas (IS), and work to juxtapose, combine, and formalize the two [10, 11, 12, 7, 13, 14]. We treat objects in the system as Object image schemas, or as picture producers (known as PPs in conceptual dependency).
The conceptual analyzer system performs primitive decomposition-based language understanding without an underlying knowledge base. Within this context it is important to point out that the conceptual analyzer currently has no knowledge about objects other than their abstract spatial relationships. It knows nothing about the sizes and shapes of objects, or states of matter. For example it does not know that the earth is a much larger object than a drop of rain, that rain is a liquid, or even that the earth is spherical. As a result, the diagrams in this paper show objects simply as circles of equal size in two dimensions to convey the way in which the representations are agnostic of many of the commonly understood characteristics of the objects.
We extend previous work [8] by using part-whole relationships to specify the parts of objects that are relevant to their perceived orientations. This now allows us to, for example, identify that certain kinds of objects that are not perfectly symmetrical have tops and bottoms, left and right parts, fronts and backs, and dorsal and ventral parts. These correspond to the Part-Whole image schema. CD does retain a primitive called PART that is not one of the eleven primitive “acts” but is present in many CD conceptualizations to specify that one object (a picture producer or PP in CD) is a part or sub-object of another object (or PP). To operationalize this feature, PartOf(A,B) indicates that object A is a part of or is a sub-object of object B. In diagrams, we show parts of objects as regions of a circle representing the full object in two dimensions.
In prior work [8] the Mental Motion Pictures conceptual analyzer decomposed meanings of words such as “rise”, “fall”, “above”, and “below” in ProPara paragraphs in terms of a positional graph with edges representing the relative distance of objects with respect to the center of the earth. For example, if an object A is at a position that is greater in altitude than another object B, this relationship is represented as an edge between A and B in the position map indicating that A is further away from the center of the earth than B. Words for movement such as “rise” and “fall” are represented by instantiating a CD PTRANS (corresponding to the Source_Path_Goal image schema) and instantiating the center of the earth as an object in the “to” and “from” cases in a CD PTRANS conceptualization. The center of the earth was chosen as the reference point because many of the input texts involve events and object movements both above and under ground.
In the current proposal, we expand the position map idea so that it incorporates relationships between any collection of objects, not only relationships between objects and the center of the earth. We introduce a new primitive, InBetween(), to represent how any three objects are posed with respect to each other in space. For three Objects or PPs, A, B, and C, InBetween(A,B,C) indicates a situation in which B is between A and C. This primitive could be used to represent altitude or changes in altitude relationships. For example InBetween(A,B,CenterOfEarth) could be used to indicate that Object B is closer to the center of the earth than Object A, and thus A is at a greater height or altitude, just as with the positional map in earlier work. In the spirit of a first approximation that maximizes the abstractness of the primitive definition, we assume for now that there exists a straight line which passes through all the objects in an InBetween().1
Combining the more general InBetween() spatial relation primitive with Part-Whole relationships allows us to propose representations not only of where objects are located in space with respect to one another, but also represent how they are oriented or how they are facing or not facing one another. Figure 1 illustrates how this works for the orientation expressions “right-side up” and “upside-down” in both pictorial form and using PartOf() and InBetween().
The representation identifies certain inherent parts of an object, its Top and Bottom, using the PartOf() primitive (for the Part-Whole image schema or PART CD primitive). PartOf(Top, Object) and PartOf(Bottom, Object) indicate that Top and Bottom are part of an Object called Object. Then, since Object parts are also Objects in their own right, InBetween(Top, 1For ProPara paragraphs describing events such as rain falling from a cloud and landing on the ground, or rays of light traveling from the sun and being absorbed by a leaf, the generality lost due to this assumption appears to be not very consequential. Bottom
Bottom, CenterOfEarth) is used to indicate that the Bottom part of the object is lower or closer to the center of the earth than the Top part, and thus it must be oriented right-side up (Figure 1, left). If the same object parts are identified and, instead, InBetween(Bottom, Top, CenterOfEarth) were the case, then the Top part of the object is lower or closer to the center of the earth than the Bottom part, and it must be oriented upside down (Figure 1, right).
In a second pair of examples, Figure 2 illustrates how this works for the orientation expressions “facing” and “prone” in both pictorial form and using PartOf() and InBetween(). The representation identifies the Front and Back of an object as its inherent parts using the PartOf() primitive. PartOf(Front, Object) and PartOf(Back, Object) indicate that Front and Back are part of an Object called Object. Then InBetween(Back, Front, TheSun) is used to indicate that the Front part of the object is lower or closer to the sun than the Back part, and thus it must be oriented facing the sun (Figure 2, left). If the same object parts are identified and, instead, InBetween(Back, Front, CenterOfEarth) were the case, then the Front part of the object is lower or closer to the center of the earth than the Back part, and it must be oriented in what is commonly referred to as a “prone” position (Figure 2, right).
Related work has examined using image schemas and primitive decomposition systems for knowledge representation, commonsense reasoning systems, and knowledge bases [11, 15, 16], and using crowdsourcing to build knowledge bases of image schemas and conceptual dependency structures [13, 10, 12]. Other work has drawn comparisons between image schemas and conceptual dependency primitives [7], and has formalized CD by modeling conceptual dependency primitives with image schema logic [14]. Primitive decomposition systems have
Front The Sun
“Facing the sun” PartOf(Front, Object) PartOf(Back, Object) InBetween(Back,
Front, TheSun)
Back Front also been proposed for enhancing self-supervised learning [17]. Recently there have been applications of image schemas and image schema logic to autonomous and reactive robotics [18, 19].
In this brief paper, we present an informal proposal for novel primitive decompositions of positions, spatial relationships, and orientations of objects in space. We introduce a spatial primitive which represents that one object is positioned in between two other objects, and combine it with part-whole object relationships to decompose concepts corresponding to commonly used terms for orientations of objects in their environments. Having these kinds of abstract primitive decompositions will enable rich representations through the variety of ways in which they can be combined with themselves and still other primitives. We look forward to applying the idea to natural language understanding and generation systems based on componential analysis and primitive decomposition. [5] S. L. Lytinen, Conceptual dependency and its descendants, Computers & Mathematics with Applications 23 (1992) 51–73. [6] R. C. Schank, C. K. Riesbeck, Inside Computer Understanding: Five Programs Plus Miniatures, Lawrence Erlbaum Associates, Hillsdale, NJ, 1981. [7] J. C. Macbeth, D. Gromann, M. M. Hedblom, Image schemas and conceptual dependency primitives: A comparison, in: Proceedings of The Joint Ontology Workshops, Episode 3: The Tyrolean Autumn of Ontology, The International Association for Ontology and its Applications, Bolzano-Bozen, Italy, 2017. [8] M. Zhou, B. Duah, J. C. Macbeth, Novel primitive decompositions for real-world physical reasoning, in: K. R. Thórisson (Ed.), Proceedings of the Third International Workshop on Self-Supervised Learning, volume 192 of Proceedings of Machine Learning Research, PMLR, 2022, pp. 22–34. URL: https://proceedings.mlr.press/v192/zhou22a.html. [9] B. Dalvi, L. Huang, N. Tandon, W.-t. Yih, P. Clark, Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 1595–1604. [10] J. C. Macbeth, M. Barionnette, The coherence of conceptual primitives, in: Proceedings of the Fourth Annual Conference on Advances in Cognitive Systems, The Cognitive Systems Foundation, Evanston, Illinois, 2016. [11] J. C. Macbeth, Conceptual primitive decomposition for knowledge sharing via natural language, in: Proceedings of The Joint Ontology Workshops, Episode 3: The Tyrolean Autumn of Ontology, The International Association for Ontology and its Applications, Bolzano-Bozen, Italy, 2017. [12] J. C. Macbeth, S. Grandic, Crowdsourcing a parallel corpus for conceptual analysis of natural language, in: Proceedings of The Fifth AAAI Conference on Human Computation and Crowdsourcing, AAAI Press, Quebec City, Canada, 2017, pp. 128–136. [13] D. Gromann, J. C. Macbeth, Crowdsourcing image schemas, in: Proceedings of The Fourth Image Schema Day (ISD4), The International Association for Ontology and its Applications, Bolzano-Bozen, Italy, 2018. [14] J. C. Macbeth, D. Gromann, Towards modeling conceptual dependency primitives with image schema logic, in: The Fourth Workshop on Cognition And OntologieS (CAOS IV) at The Fifth Joint Ontology Workshop (JOWO’19), The International Association for Ontology and its Applications, Graz, Austria, 2019. [15] E. Cambria, Q. Liu, S. Decherchi, F. Xing, K. Kwok, SenticNet 7: A commonsense-based neurosymbolic AI framework for explainable sentiment analysis, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 3829–3839. URL: https://aclanthology.org/2022. lrec-1.408. [16] S. De Giorgis, A. Gangemi, D. Gromann, Imageschemanet: formalizing embodied commonsense knowledge providing an image-schematic layer to framester, Semant. Web J.(2022, forthcoming) (2022). [17] J. C. Macbeth, Enhancing learning with primitive-decomposed cognitive representations, in: Proceedings of The First Annual International Workshop on Self-Supervised Learning (IWSSL-2020), Cambridge, MA, 2020. [18] K. Dhanabalachandran, M. M. Hedblom, M. Beetz, A balancing act: Ordering algorithm and image-schematic action descriptors for stacking objects by household robots, in: The Eighth Joint Ontology Workshops (JOWO22), RobOntics: 3rd International Workshop on Ontologies for Autonomous Robotics Jönköping, Sweden, August 15-19, 2022, CEUR-WS, 2022. [19] M. Pomarlan, S. De Giorgis, M. M. Hedblom, M. Diab, N. Tsiogkas, Thinking in front of the box: Towards intelligent robotic action selection for navigation in complex environments using image-schematic reasoning, in: The Eighth Joint Ontology Workshops (JOWO22), RobOntics: 3rd International Workshop on Ontologies for Autonomous Robotics Jönköping, Sweden, August 15-19, 2022, CEUR-WS, 2022.