Domains that rely heavily on robotics have shown increasing interest in collaborative, intelligent, and reasoningcapable autonomous systems. In pursuit of these interests, a standard method for increasing autonomy among robotic agents and enabling collaboration is the use of knowledge representation in the form of ontologies and knowledge graphs. To further enhance these systems, we are creating a modular ontology for autonomous robotic orchestration, using the Modular Ontology Modeling methodology, which focuses on developing ontology patterns to facilitate reuse. This efort aims to achieve a greater goal, an artificially intelligent orchestrator capable of commanding autonomous robots in unpredictable environments, titled Task Adaptation with Shared Knowledge for Multi-Agent Teaming Systems (TASK-MATS). We introduce well-known robotic architectures that incorporate specific ontologies into their frameworks and discuss how those foundational ontologies inspired an expansion of their concepts to fit our use case. To conclude, we briefly discuss the next steps in our ontology modeling eforts.
Domains that rely heavily on robotics have shown an increasing interest in autonomous systems that are
collaborative, intelligent, and capable of reasoning. In pursuit of these interests, a standard method for
increasing autonomy among robots and enabling collaboration is the use of knowledge representation
in the form of ontologies and knowledge graphs (KGs) [
Considering these aspects for the current ontological representations of complex autonomous systems,
we want to engineer an ontology that could enable these multi-agent systems (MAS) to be governed
and commanded by an AI Orchestrator (AIO) capable of the following: understanding its environment,
decomposing goals into atomic subtasks, agent assignment based on capabilities, and status verification.
This is part of a greater efort to create a KG-powered MAS, titled Task Adaptation with Shared
Knowledge for Multi-Agent Teaming Systems (TASK-MATS). More importantly, it would be designed
to support reusability and allow for adaptation to the domain of need. This ontology would be modeled
using state-of-the-art (SOTA) best practices to ensure modularity and reusability through the use of
ontological design patterns (ODPs) [
The rest of this paper is organized as follows. Section 2 showcases how ontologies and KGs are being utilized to enable autonomous robots. Section 3 showcases the current results from the eforts being made to actualize the proposed ontology. Finally, in section 4, we conclude and discuss the future direction of our work.
Bestowing robots with autonomous capabilities in dynamic unstructured environments has proven to be a momentous challenge. However, significant headway has been made toward achieving this through the utilization of robotic frameworks that employ ontologies to aid in building an autonomous agent’s worldview.
Olivares-Alarcos et al. [
KnowRob 2.02 is arguably one of the more successful implementations of a framework for enhancing
robotic agent capabilities in unstructured environments3 [
KnowRob 2.0 is significantly more complex than other existing frameworks, as it includes several
unique architectural components, such as virtual simulations, episodic memory, and the ability to learn
from their outputs. However, to enable any of these to work, an ontology is required to represent the
semantic meaning of the data in the knowledge base. Without their ontology, the robots that employ
their software would be unable to infer or reason about their environment [
The Modular Ontology Modeling (MOMo) methodology is described in detail in [
Before we discuss the paper’s position, we summarize our
implementation of [
Our approach toward creating a modular ontology for robotic orchestration followed the methodology
presented and outlined in MOMo. This requires that we begin by identifying a use case, determining
useful datasets, and developing competency questions that would allow for future verification of the
ontology. We reiterate our use case: to construct an SOTA reusable ontology that allows for complex
MAS that incorporates an AIO for task assignment and verification in dynamic and unstructured
environments. From this, we identified many applicable datasets that encompass data on objects, robotic
manipulation, environments, tasking, and more. We then began identifying key notions, or rather,
concepts that overlap within the competency questions and data [
The key notions act as the basis for modeling the ontology. First, we examined patterns that already
exist within MODL [
The deliverables for this ontology and the current collection of identified datasets are available in
our GitHub repository7.
3.1. Goal & Task
Task decomposition is and remains a central theme across many prior works. Several frameworks, such
as KnowRob 2.0 [
Building on these past limitations and lessons observed, we developed a Goal-Task module to formalize how high-level objectives can be decomposed into actionable units of work within our orchestration ontology. Following the principles of MOMo, we treat Goal as the abstract representation of the desired outcome while keeping Task as the minimal operational unit required to achieve that outcome, keeping it atomic and hence not further decomposable.
As portrayed in Figure 2, a Goal instantiated relationships to Task through hasTask property for explicit decomposition of work from strategic intent to an actionable executable item. This modeling approach explores a diferent angle of decomposition and helps preserve the work decomposition hierarchy from a ‘what needs to be achieved and done’, rather than generic breakdowns, which focus on the ‘how it should be done’, to capture the broader semantics of intention behind work planning and completion. This also captures the Spatial and Temporal constraints (hasDeadline, isWithinBounds and hasOperationalExtent), which contextualize the broader tasks and goals of a system within real-world limits, also bounding it to their respective Environment extents. Each Goal is further linked to a success condition via hasSuccessState, allowing the AIO to evaluate completion criteria, especially in dynamic environments.
Similarly, individual Tasks were modeled for sequencing for partial order execution and dependency management (dependsOnTask and hasNextTask), as well as references to required objects (requiresObject) and necessary Archetype (reEqxutierensdAinrcghetsyepvee)r.al pat- Figure 2: The schema diagram for Goal and Task. The yellow boxes terns from MODL, these indicate the classes, the blue dashed boxes refer to external design primitives, unlike schema entities and the yellow ellipses indicate the datatypes several prior frameworks tied to those classes. Open arrows indicate SubclassOf relathat embed semantics with tions. 7The current work for this ontology https://github.com/kastle-lab/Autonomous-Robotic-Orchestration-Modular-Ontology. (a) The schema diagram for Capability.
(b) The schema diagram for Archetype.
static rule sets, ensure
compatibility with other modules for ontological consistency across the framework. Modeling was
strongly influenced and grounded in task dependencies, temporal ordering and goal hierarchies by
datasets such as ALFRED [
Both the OCRA and KnowRob frameworks import a Capabilities class as defined by SOMA. However, neither of these ontologies extends or makes use of the class in a meaningful way. Notwithstanding, a well-developed Capability class is required for our use case. In SOMA, the Capability class is entirely abstract, but further extending it would be more appropriate for a MAS with an AIO for tracking the set of capabilities of each robotic agent under its influence. The purpose is to provide the AIO with the capacity to reason about the available agents and their capabilities under its authority. To then appropriate the necessary agents to achieve an overall goal and complete atomic-level tasks, based on the required skills. The current schema for the capability pattern is shown in Figure 3a.
Furthermore, the modeling of the capability pattern was heavily influenced by existing available data. One useful dataset of note is RH20T-P, provided by an internationally collaborative research team from China and Australia. In their dataset, they model what they define as Primitive Skills under two categories: Gripper-Based and Motion-Based [24]. These are useful distinctions to make in the Capabilities class, as the capacity of robotic agents is typically constrained by their hardware and software. The composition of such capabilities results in the next class of discussion, Archetype.
Analysis of Figure 3a reveals a connected class, Archetype, through the relationship requiresCapability, as exhibited in Figure 3b. The Archetype class is a thematic representation of an agent’s capability. For example, a robotic Agent of the Explorer Archetype would have specific sensor and motion-based capabilities. We chose to model an Archetype class rather than use the nomenclature Role, because there are important distinctions between the two. While Role is typically used to classify an agent’s immediate characteristics, the Archetype class is instead more representative of the culmination of all the characteristics an agent has based on their capabilities (i.e., a set of skills). The specific set of those capabilities gives an agent an archetype, allowing it to fill many roles until it is assigned to an immediate
task, where it will fulfill a specific role. As these patterns become more concrete, the archetypes would be enforced through the constraints imposed by axioms.
We are engineering a modular ontology, using SOTA best practices, to enable complex autonomous MAS to include an AIO capable of managing other robotic agents. Currently, there are two well-known exemplary frameworks, OCRA and KnowRob, that showcase the significant role that ontologies play in the pursuit of such complex systems. These frameworks use the same foundational ontologies — DOLCE+DUL and SOMA — and extend them as needed. In our examination of these, we found it necessary to extend and add to some of the concepts presented in the aforementioned ontologies, not necessarily an extension of the ontologies themselves, by creating the ODPs we identified as a requirement for our use case. This is part of a greater efort to create a KG-powered MAS, which we are coining as TASK-MATS.
Future Work We have identified several directions for future work. Of course, we must finalize our own ontology eforts. As a part of this, however, an exhaustive collection of specification data, especially related to capability, must be executed. This ensures that not just an ontological notion is well-modeled, but also that it is empirically validated using real-world scenarios. The TASK-MATS project supports this initiative, but annexing or otherwise integrating into the wider community is a must. Finally, we must look forward into how this ontology can be used in broader neurosymbolic frameworks, incorporating planning and multi-modality. Though the work is mostly preliminary and much future work remains, we conclude that, at the very least, the current state of this work is a proper step in the direction toward achieving our endeavors.
Michael McCain and Cogan Shimizu acknowledge support from DAGSI RX26-24: “TASK-MATS: Task Adaptation with Shared Knowledge for Multi-Agent Teaming Systems”.
During the preparation of this work, the author(s) used Grammarly to perform a grammar and spelling check. Generative AI was used to identify academic resources during the literature search phase of this research, but these sources were manually reviewed. on Computer Vision and Pattern Recognition (CVPR), 2020. URL: https://arxiv.org/abs/1912.01734. [24] Z. Chen, Z. Shi, X. Lu, L. He, S. Qian, H. S. Fang, Z. Yin, W. Ouyang, J. Shao, Y. Qiao, C. Lu, L. Sheng, Rh20t-p: A primitive-level robotic dataset towards composable generalization agents, arXiv preprint arXiv: 2403.19622 (2024).