Dealing with robotic actions in uncertain environments has been demonstrated to be hard. Many classic planning approaches to robotic action make the closed world assumption, rendering them inefficient for everyday household activities, as they function without generalizability to other contexts or the ability to deal with unexpected changes. In contrast, humans robustly execute underspecified instructions in unfamiliar environments. In this paper, we initiate our research program where we propose the use of functional relations in the form of image-schematic micro-theories, formally represented in ISL , to enrich action descriptors with semantic components. It builds on the body of work in embodied cognition showing that human conceptualization of action sequences is founded on abstract patterns learned from physical experiences in the form of spatiotemporal relationships between object, agents and environments. These theories are used to inform action selection mechanisms for behavioral robotics written in EL++ and we argue how these micro-patterns can be applied in a more general way to deal with underspecified action commands and commonsense problem-solving.
Robot agents are starting to accomplish human-scale everyday manipulation tasks such as setting a
table, cleaning up, and preparing (very) simple meals. Most knowledge representation approaches
to reasoning about actions conceptualize the repertoire of robots as a state transition system, with
actions as atomic transitions between states [
This abstraction is critical from the robot agent perspective because the main reasoning task of a robot agent is to infer how it has to move its body in order to accomplish an underdetermined task such as “put the oat milk on the table” without causing unwanted side effects. However, the use of abstract action models make robots inflexible as the same task can be implemented through different body motions: the robot can close a door with its hand but, analogically, it can also use its elbow if the hand is occupied by carrying an object. Further, the same motion can implement different tasks: a pushing motion can close a door but also position an object more accurately. The lack of such reasoning capabilities not only limits the manipulation tasks that robots can perform, but the accuracy by which it performs the actions as well.
There are approaches to include more detailed representations, such as the combination of task
and motion planning. However, motion planning typically tries to compute collision-free paths
rather than motions that achieve certain effects [
In this paper, we propose to equip knowledge representation and reasoning systems for robotic agents with a generalizable layer of understanding of the conditions in the environment. This allows the robot to reason about action execution in terms of motion types when encountering unexpected changes in a given situation. For any physical state of affairs in the world, this layer of understanding can be described using a set of functional relationships between objects, agents and environments, called image schemas.
While in an early stage, we will argue that the contribution of our proposition is multifold: (i) Reasoning about functional relationships: It allows robots and artificial agents to reason about the functional relationships between objects and other entities situated in complex environments in a cognitively-plausible way. (ii) Reasoning about alternatives to a plan: This semantic grounding of the environment offers problem-solving capabilities in that it provides means to expand the reasoning outside the specified action plan. This enables corrective actions such as avoidance of blockages. (iii) Increase adaptability through analogy: It also offers generalizability in that information relevant to one situation can be analogically transferred to another situation. (iv) Improve natural language understanding: Building on research founded in cognitive linguistics, it enables a large body of work to be leveraged towards increasing natural language understanding and for robots to follow human instructions and commands more accurately.
Adult humans are creatures of habit. Repeated experiences with similar situations, i.e. particular
states of the world, shape us into experts on a variety of situations possible in our environment.
We can use generalized patterns of information from our previous experiences to reason about
outcomes of uncertain situations. These expectations extend to action instructions as well.
Asking someone to set the table or get the milk will likely result in a satisfactory outcome,
regardless of whether that person has been in that particular kitchen or not. Humans have
extensive understanding of the affordances of a kitchen, which tools are needed for eating, and
an appropriate set-up of a dining table given different contexts, such as type of meal or number
of participants. In comparison, robots, which are relatively proficient in well-defined tasks
in constrained environments [
To aid this research agenda, we propose to utilize the underlying patterns of expectations
found in humans as suggested by the theory of embodied cognition [
One way this could be computationally realized is to characterize the image schemas in relation
to the affordances they require [
While affordances can be described by the interplay of respective dispositions of objects and
agents [
The image schemas capture relationships such as SOURCE_PATH_GOAL (SPG) - depicting
movement of an object between two points, LINK - the force-dynamic relationship that connects
objects with one another, and VERTICALITY - vertical movement, and relative position and
symmetry on the vertical axis. Another strength of using image schemas for formal research on
underspecified environments and instructions is that they have been shown to come in different
levels of specificity, e.g. the difference between a tight and loose containment, or how SOURCE_
PATH_GOAL ranges from simple object movement to increasingly specific notions with source
and goal locations [
The embodied grounding of image schemas makes them a prime subject to be learned through
statistical methods and deep learning. For example, in the work by [
The temporal dimension is based on linear temporal logic (RTL) over the reals [
::“ | J | ␣ | ^ | U | where U reads as “ holds, until ” and reads as “ holds, since .” As it is standard in temporal logic, we can define additional temporal operators based on these, for instance, operators like: F (at some time in the future, ) is defined by JU ; and, G (at all times in the future, ) is defined as ␣F␣ ,
In ISL , the temporal structures, often disregarded due to the increase in complexity in
formal image schema modeling, constitute the primary model-theoretical object, e.g., a linear
order to represent the passage of time, in which complex propositions that employ a secondary
semantics are included. The atoms are topological assertions about regions in space using RCC,
the relative movement of objects with respect to each other using QTC, and relative orientation,
using CD. We refer the reader to [
ISL axioms are based on a concept language in First Order Logic (FOL), making it an expressive tool to represent different situations and concepts. The idea is that each image schema (e.g. LINK (x,y) and SOURCE_PATH_GOAL (x,p,s,g)) is modelled using the logic’s semantics and is represented using FOL.
While ISL is predominantly a modeling language, the image schemas are defined by internal logical rules that can be used to reason with. For instance, the CONTAINMENT relationship is transitive in that an object that is contained within another object will move if the container moves (consider how the milk will remain in the carton when the carton moves), see: @, :, , :, : ℎ P : pCONTAINED_INp, q ^ SPGp, , , qq Ñ SPGp, , , q Following the same reasoning, a LINKed relationship ensures that what happens to one of the objects is transferred also to the linked object (a robot holding a carton will ensure that if the robot moves, then the carton moves as well). Likewise, a SUPPORTed object will move, if the SUPPORTing object is transferred to another location (consider a robot carrying the milk-carton on a tray).
These kinds of built-in rules for our expectations of the environment offer the possibility to make predictions for the outcomes of particular actions. These rules also offer the possibility for dealing with unexpected problems that can arise. For instance, if the robot’s movement is BLOCKed by another entity getting in the way, the robot would be able to reason if that object is in movement and thus, wait for it to pass, or if it is still, circumvent the BLOCKAGE. Likewise, a robot would be able to reason about how a container needs to be opened for something to be able to exit the container. The next section will tackle this and related reasoning challenges.
One of the most common things we ask other people to do is to fetch different things for us. From
a household robot’s perspective, the instruction Get the oat milk can be described as a particular
instance of a Fetch-And-Place action descriptor. In addition to the call to perform the action, the
instruction includes additional important – albeit implicit – forms of knowledge. First, the noun
oat milk evokes information about what it is, e.g. a perishable good, where it is usually stored
and so on. The SOMA ontology of everyday activities [
@, :p pq Ø CONTAINED_INp, q ^ pqq Second, the verb get requires the understanding of transporting something by evoking a SOURCE_ PATH_GOAL schema with the locations of the source and the goal being an integral part of the transportation expressed. The ABox for the oat milk reveals that p q : , and the robot needs to understand that the source of the instruction (the person speaking) reveals the goal of the transportation – namely, close to the speaker, p q : .
If the robot has an episodic memory concerning oat milk, in addition to the ontological semantics contained in a NEEM, it has information that milk is a perishable liquid stored in a container that is placed on a shelf in a fridge. Image-schematically this represents nested CONTAINMENT: the object in question is inside one container (carton) which in turn is inside another container (fridge) for different purposes. This is crucial information for knowing how to treat the object. While a robot could theoretically move the entire fridge to the person asking for oat milk, this is an inefficient way of solving the problem. Likewise, it is not very smart to take the oat milk out of the carton before attempting to moving it. One reason for this is because the purposes for CONTAINMENT are fundamentally different, the liquid needs tight CONTAINMENT for transportation, whereas the fridge’s loose CONTAINMENT has nothing to do with transportation but instead for static storage and preservation.
The second image-schematic component is the movement of the oat milk from the fridge to the person asking for it. This represents the construction of SOURCE_PATH_GOAL capturing different levels of specificity of the conceptualizations of movement. The classical linguistic interpretation is that a trajector (object or agent) moves along a path from a particular SOURCE to a determined GOAL. In this case, the robotic agent needs to be able to deduce that the SOURCE is the initial location inside the fridge and that the GOAL is to reach the near vicinity of the person asking for the milk. This may sound like a trivial problem, but it includes not only the SPG schema, but also the CONTAINMENT schema as the Going_OUT schema is part of the schema’s dynamic relationships and can, in isolation to the whole event, also be described as a combination of the image schemas SPG and CONTAINMENT.
The third image-schematic relationship we cover in our working example is SUPPORT. Obviously, all objects are supported by the ground, but for a robot to masterfully be able to manipulate objects, it is not possible for it to neglect the naïve rules and structures of placing things on top of other things. With the oat milk, this offers important information to be transferred from the source to the goal state. At the source inside the fridge, the oat milk is vertically2 SUPPORTed on a shelf. This could be seen as a required property of the oat milk-carton throughout the action and in the goal state, as it has an opening at the top.
In ISL , most3 image schemas and their hierarchical graphs can be formally represented in the form of ontological patterns. Any formalization using ISL would use their placeholder names to access the full axiomatization. For instance, the LINK, image schema is formalized using p, q ^ p, q ^ p, q describing how for two objects, x,y, to be linked, they are externally connected (asserted by the RCC8 operator EC) and there is a force from each respective object towards the other. The action event can be described as six different image-schematic states, depicted and verbalized in Figure 1.
The purpose of understanding image-schematic relationships in scenes is not only to identify
the logical axiomatizations thereof in ISL , it is to describe the underlying patterns for
understanding the meaning of events and actions. By giving an artificial agent access to this
level of semantic layer, it becomes possible to reason about this particular scenario, as well
as to transpose this knowledge to other similar situations with different objects and contexts.
Additionally, this type of reasoning becomes vital to recover from errors and mishaps that might
occur[
2Assuming that the carton is elongated on the vertical axis.
3The logic is not able to elegantly handle transformational relationships involved in image schemas such as Spiraling or Scaling, as they are more loosely described in terms of other objects and instead their previous states. For this the addition of mathematical functions could be a way forward.
Scene 1 Scene 2 Scene 3 Scene 4 Scene 5 Scene 6
Scene 1: Initial state. There is a LINK between the Handle and the FridgeDoor. The FridgeDoor is in CONTACT with the Fridge. The Milk is CONTAINED_INSIDE in the Carton, the Carton is VERTICAL and SUPPORTed by the Shelf which is CONTAINED_INSIDE the Fridge.
Scene 2: Opening the door. To open the door, the FridgeDoor needs to leave its state of being in CONTACT with the Fridge. This is done by SOURCE_PATH, where the Handle moves along a CircledPath away from the Fridge.
Scene 3: Lift milk-carton. Through VERTICAL oriented SOURCE_PATH_GOAL the SUPPORT from the Shelf on the Carton is removed, since the Milk is CONTAINED_INSIDE in the Carton it MOVEs as well.
Scene 4: Take the Milk out. The Carton goes OUT of the Fridge through its opening, and is no longer CONTAINED_INSIDE.
Scene 5: Milk goes to its destination. Since the Carton MOVEs towards the Table, the Milk MOVEs to the Table.
Scene 6: Goal state of the Milk resting on the Table: The Carton’s movement is BLOCKed when it comes in CONTACT with the Table, turning the force from the movement into the reversed force as SUPPORT from the Table.
In this section, we will illustrate how ISL can contribute to robotics to address the issues
listed in the introduction. We will, for each item in turn, explain the underlying problems, provide
an example of reasoning in ISL to address such problems, and then show how the results
inform the construction of axioms in a simpler formalism, EL++ [
One thing that we have repeatedly stressed in this paper is how the image schemas offer a cognitively plausible method for artificial agents to reason about their surroundings. A simple example is how humans, birds, and other animals understand that if you want to eat a nut you have to crack the shell before you can take it out. Likewise, humans are quite experienced with the understanding that if you want to take the milk out of the fridge, it is not possible unless you ifrst open the fridge door.
This level of commonsense reasoning is intuitive in biological intelligence, but it has been demonstrated to be time-consuming to axiomatize the full scenarios (as with the egg cracking problem) and inefficient to rely purely on statistical methods. This is where the image schema logic could play a vital role.
To give an example of how a CONTAINMENT inference can work in ISL , consider the following axiom about solid objects (for simplicity, we omit to include conditions about no parthood relations between the objects involved)4:
@1, 2 : 1 ‰ 2 Ñ ␣ p1, 2q ISL allows us to define a few regions of interest around an object: its interior (which we will take here as a primitive predicate), its exterior, and openings. Let then a closed container be an object with an interior but no openings, and an enclosed object be one contained inside a closed container.
p, q :“ p, q ^ “ p Y q p, q :“ p, q ^ p, q
^ p, q ^ p, q pq :“ ␣D p, q pq :“ D pq ^ CONTAINED_INp, q Reasoning inside the RCC8 fragment of ISL then establishes that, for solid objects, to touch an enclosed object without also being enclosed in the same container is impossible: @, , : p, q ^ ␣p, q Ñ ␣p, q In upcoming work, we intend to use EL++ ontologies to encode image-schematic knowledge for action selection. The previous result from ISL may be approximated in EL++ as: ” D.
[ Ď K [ Dpℎ .q
[ pDℎ . q Ď K Meaning, a relationship in which one participant is a free object and another is enclosed cannot be of type EC. This has consequences for action selection because, e.g., an EC relation between a gripper and an item to grasp is necessary. If such a relation is impossible, then reaching for an item should be delayed. Further axioms could pinpoint possible alternative actions, such as to manipulate handles on the container to open it.
Another vital contribution that the image schemas bring to reasoning about action execution is dealing with unexpected situations. In a kitchen, it is not unlikely that multiple humans are 4As defined in RCC8: - Partial Overlap; - Externally Connected. - Proper Part, - set theoretical complement in R3. present at the same time and may continuously change the state of the environment. If a person at a table asks the robot to fetch them the milk, the robot needs to be able to reason about any changes that might have taken place. Perhaps the originally intended path is no longer possible to take because another person put something in front of the robot or is actively crossing the path in that instance. Not only it is expected that the robot should stop and not run over the other human, but it should also be able to redirect its movement if the object on the path is unlikely to move anytime soon, i.e. if it does not have a movement state. If the human blocking its movement is simply walking past as part of an SOURCE_PATH_GOAL of its own right, then the robot can simply take a break before continuing along the same path. However, if the person, or item, remains on the path, the robot needs to be able to reroute to still be able to successfully reach the goal of the instructions. It would do this, by generating a new SOURCE_PATH_GOAL construct in which the source is no longer based at the fridge, but at the location of the BLOCKAGE, and the path is no longer the fastest route (or what previously had been suggested) but a route that bypasses the blocking object.
The above discussion can be formally represented in ISL as follows with omitting the assumption that the source and goal (S,G) are TPP, tangential proper parts, of the path. Assuming again the axiom of solid objects, and the following axiom about SPG:
SPGp, , , q ^ Gp p, qq Ñ Fp p, qq That is, if a trajector follows a path , it will pass through any region along the path. Then, one can show the following: ‰ ^ Gp p, qq Ñ Gp␣SPGp, , , qq In other words, if a path is forever blocked, it is never possible to use it to go from S to G.
As before, we wish to provide a robotic agent with some simple rules to select, or filter out, actions. The previous result from ISL could be approximated in EL++ thusly, assuming appropriate recognition procedures for entities such as paths and stationary objects, and appropriate controllers for actions such as moving the robot base: ℎ ” ℎ [ D.
[ D. ℎ Ď K That is, no action that involves moving the base should use a blocked path. If such a path is the one currently used by the robot, it should search for a different one. A similar set of simple rules might encode for the robot that it may be worth waiting if a path is blocked by a moving object: ℎ ” ℎ [ D. [ D. ℎ Ď Such a set of action rules would be justified if one believed that objects moving away from a path eventually do not overlap it:
Gp Ðâ q Ñ Fp␣ p, qq
Bypassing things like BLOCKED_MOVEMENT by redirecting routes requires the robot to rethink its actions based on new image-schematic states of the world. This is useful, but it is also possible to use this in a more general way by abstracting away from the actual parts of the world.
This builds on the idea of analogical reasoning, that there are underlying patterns that can be transferred from an information rich source domain to an underspecified target domain. For robotic actions, this offers the possibility to reuse previously learned relationships. One crucial component for successful analogical transfer is that the source and target share the same structure. In the settings of functional relations as the foundation to guide robotic action selection, these patterns are also useful as a basis for generalization. For instance, if a robot has access to the image-schematic information that for something to be taken out of a closed container as for instance a fridge, it can use this information to reason about similar CONTAINMENT situations. It can use this generalized information to take the lasagna out of the oven (take note of how Figure 1 would be exactly the same with this example), a letter out of a envelope and join the masses of biological species whose evolutionary predecessors learned long ago that the nut needs to come out of its shell.
A less complex analogy, as only one object needs to be exchanged, is how it is possible to close a door with another body part than a hand, should it be preoccupied holding milk-cartons or lasagnas. In this case, what we want is to describe, image-schematically, what it means for a solid object to push another, via some third solid object: ℎp, , q :“ p “ _ LINKp, qq ^ p ù q ^ p, q A robot using EL++ rules to select and parameterize actions might then be interested in classifying what objects it can push with: ℎ Ď D. ℎ Ď ℎ
D . Ď ℎ
Another important requirement of a successful household robot system is to be able to understand
human instructions. Natural language instructions are usually underspecified and contain vast
amounts of ambiguity, polysemy, and implicit information that need to resolved and explicated in
order to execute the corresponding actions appropriately. Another problem is that instructions
often omit vital semantic components such as determiners, quantities and even the object
themselves [
To improve natural language understanding in robotics, with a special focus on instructions,
we employ an efficient construction-based parser [ 31] that produces semantic specifications as
connected RDF triples that represent as much of the meaning of the instructions as contained
in the textual commands. All terms in these semantic specifications are aligned to the
SOMASAY module [32] that is part of the larger SOMA framework [
Using commonsense reasoning to improve robotic action selection is not a novel idea, it has
been a fundamental component since the beginning of formal research on intelligent systems (a
comprehensive overview is given in [35]). Many researchers (e.g. [
Similar to the ideas in this paper, is the work by [38]. They consider activity knowledge as a means to fill the gaps in abstract instructions, but treat these gaps in a much more general point of view than the specifics found in image schemas. A more general approach to activity modeling for robotic agents is presented by the IEEE-RAS working group ORA [39]. The group has the goal of defining a standard ontology for various sub-domains of robotics, including a model for object manipulation tasks. It has defined a core ORA ontology [ 40], as well as additional modules for industrial tasks such as kitting [41]. In terms of methodology, we differ in the foundational assumptions we assert, with important consequences on the structure of our ontology, modeling workflow, and inferential power. In the case of ORA, the SUMO upper-level ontology is used as foundational layer. Compared to SUMO, we use a richer axiomatization of entities on the foundational layer, and put particular emphasis on the distinction between physical and social activity context.
Unlike most previous methods, that often build action descriptors for particular actions and
scenarios, we suggest relying on the generalized information learned from the sensorimotor
experiences, encoded as functional relationships based on image schemas. While there exists research
on how to formalize image schemas [42] and to use them for simulation-based reasoning [43],
the role they play in active applications is not quite as thoroughly investigated. Another novel
approach is to construct hybrid reasoning pipelines that connect simulation-based reasoning with
qualitative reasoning about functional relations [
At this stage, the contributions of the paper remain purely theoretical. However, the novelty of the approach and our conviction of the ideas underlying the core concepts and their contributions motivates future work.
The next steps of this research program is to further strengthen the applicability of this work by providing a more feasible connection between ISL and simple formal languages commonly used in robotics, such as EL++. Additionally, we intend to develop an image schema parser that can identify and extract image-schematic relationships from the subsymbolic data of robotic simulations and visually recorded human activity to provide automation to the system. Thirdly, we aim to connect the formal part and the identification parser to the body of work in cognitive linguistics to improve the robotic agents’ understanding of instructions in natural language. Acknowledgements The authors thank John Bateman and Fabian Neuhaus for valuable insights and constructive feedback on the paper. The research reported in this paper has been supported by FET-Open Project #951846 “MUHAI - Meaning and UNderstanding for Human-centric AI” funded by the EU Program H2020 and the German Research Foundation DFG, as part of Collaborative Research Center (Sonderforschungsbereich) 1320 “EASE - Everyday Activity Science and Engineering”, University of Bremen (http://www.ease-crc.org/). The research was conducted in sub-projects “P01 Embodied Semantics for the Language of Action and Change” and “R01 CRAM 2.0 - a 2nd Generation Cognitive Robot Architecture for Accomplishing Everyday Manipulation Tasks”. [31] V. S. Cangalovic, R. Porzel, J. A. Bateman, Streamlining formal construction grammar, in: Proc. of the ICCG11 Workshop on Constructional approaches in formal grammar, Antwerp, Belgium, 2021. [32] R. Porzel, V. Cangalovic, What say you: An ontological representation of imperative meaning for human-robot interaction, in: Proc. of the Joint Ontology Workshops, Virtual, Bozen-Bolzano, Italy, 2020. [33] A. Gangemi, P. Mika, Understanding the semantic web through descriptions and situations, in: Proc. of the ODBASE Conference, Springer, 2003. [34] J. Pfau, R. Malaka, We asked 100 people: How would you train our robot?, in: Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 335–339. [35] A. Olivares-Alarcos, D. Beßler, A. Khamis, P. Gonçalves, M. Habib, J. Bermejo, M. Barreto, M. Diab, J. Rosell, J. Quintas, J. Olszewska, H. Nakawala, E. Pignaton de Freitas, A. Gyrard, S. Borgo, G. Alenyà, M. Beetz, H. Li, A review and comparison of ontology-based approaches to robot autonomy, The Knowledge Engineering Review 34 (2019). [36] N. J. Nilsson, Shakey the robot, Technical Report, SRI, Menlo Park, CA, 1984. [37] R. D. Nielsen, R. Voyles, D. Bolanos, M. H. Mahoor, W. D. Pace, K. A. Siek, W. H. Ward, A platform for human-robot dialog systems research, in: 2010 AAAI Fall Symposium Series, 2010. [38] M. Tenorth, M. Beetz, Representations for robot knowledge in the knowrob framework,
Artificial Intelligence 247 (2017) 151–169. [39] C. Schlenoff, E. Prestes, R. Madhavan, P. Goncalves, H. Li, S. Balakirsky, T. Kramer, E. Miguelanez, An IEEE standard ontology for robotics and automation, in: IEEE Int. Conf. on Intelligent Robots and Systems (IROS), 2012, pp. 1337–1342. [40] E. Prestes, J. L. Carbonera, S. R. Fiorini, V. A. M. Jorge, M. Abel, R. Madhavan, A. Locoro, P. Goncalves, M. E. Barreto, M. Habib, A. Chibani, S. Gérard, Y. Amirat, C. Schlenoff, Towards a core ontology for robotics and automation, Robotics and Autonomous Systems 61 (2013) 1193 – 1204. Ubiquitous Robotics. [41] S. R. Fiorini, J. L. Carbonera, P. Gonçalves, V. A. Jorge, V. F. Rey, T. Haidegger, M. Abel, S. A. Redfield, S. Balakirsky, V. Ragavan, H. Li, C. Schlenoff, E. Prestes, Extensions to the core ontology for robotics and automation, Robot. Comput.-Integr. Manuf. 33 (2015) 3–11. [42] A. U. Frank, M. Raubal, Formal specification of image schemata – a step towards interoperability in geographic information systems, Spatial Cognition and Computation 1 (1999) 67–101. [43] S. Nayak, A. Mukerjee, Concretizing the image schema: How semantics guides the bootstrapping of syntax, in: 2012 IEEE Int. Conf. on Development and Learning and Epigenetic Robotics, ICDL 2012, 2012.