The task of picking up boxes and stacking them in another location is a relatively simple one for humans, however, it is particularly challenging for robots. The robot must first detect and recognize objects in its immediate environment. Following this, a task should be defined for the robot to perform. Having considered the environment, the robot will then be required to design and control a trajectory that is appropriate to its constraints. In this study, we present a semantic approach for the robotic manipulator to perceive the environment, plan the trajectory, and perform the tasks. We have designed an autonomous robotic arm ontology (ARAO) for describing the tasks and the environment in order to control robot movements. SPARQL queries running on RDF will be used to access and update the ontology model. We planned experiments on a robotic simulator that is integrated with an ontology model using the Jena API. Our preliminary findings provide us with insights into how semantic knowledge representation can be used to accomplish complex tasks via robotic manipulators in more complicated situations.
A robotic manipulator’s platform can perform a wide range of complex tasks via sequential
programming in a deterministic environment, such as those found in industrial applications. As
compared to daily life, where the environment is less dynamic and unpredictable, robotic
applications are becoming more realistic and practical. These robots are controlled by robust control
algorithms in order to ensure that they can reliably perform repetitive tasks [
In practice, the robotic manipulation process consists of two main stages in order to accomplish a task successfully. As a first step, it is necessary to identify the tasks that robots have to perform. For this purpose, it is required to perceive the environment and answer the following questions [5, 6, 7]. What information is required to achieve satisfactory results for robotic systems? What are the afordances of objects and their functions? What can robots do; by observing humans or learning through imitation? The second step is to execute the task. In general, it is a challenge to specify the type of task, its sub-actions, attributes, and associated environmental objects involved in a task. An additional challenge for robots is to specify the relation between the attributes of the object (e.g. size, shape, color) and the actions (e.g grasp, touch, push) of the manipulator [8]. Existing approaches are limited by the fact that they rely on large quantities of data and actions. Our motivation at this point is that if the tasks and environmental attributes are formally explicit, there can be benefits in terms of two aspects that contribute to the complexity of high-level task planning through robotic manipulation. First, the robot’s decision-making process, which incorporates decisions regarding the actions and attributes of the object, has enabled it to accomplish task planning [9]. The second, potentially, robots might be able to share information about tasks and environments to facilitate collaboration, allowing task verification and raising task credibility. In fact, we want to investigate how to establish linked and shared data and behavior perspectives for robotic manipulation. In other words, this study focuses on establishing the relationships between the tasks and the existing objects in the environment.
Therefore, this paper presents an approach for robots to achieve a sequence of tasks by utilizing semantic knowledge representation. An ontology that includes the definitions and relations of tasks, shapes and robots are defined. As a hypothetical validation scenario, a robotic manipulator and 3 boxes have been added to the robotic simulator to execute the tasks given in the ontology. The task sequence is mainly about reaching, grasping and releasing. The ontology helps the robot manipulator to understand the objects in the environment and their localizations. The location information of the object where the robotic manipulator will be able to perform an action is stored and updated in the ontology model. In this way, the robotic manipulator will be able to get localization information of the boxes from ontology, so that it will be able to reach the boxes and stack them at a desired location. So, with this approach that will be performed on a hypothetical validation scenario, robotic manipulator task planning will be fed from the ontology in perceiving and manipulating the environment and environment elements, supporting the robot types, and related robotic task types and capabilities.
In this section, we briefly describe the research topics and software tools that have been studied on robots using ontology approaches in the last decades. Semantic knowledge representation has been utilized in various studies in robotics. In our review of the literature, we found that many studies used Web Ontology Language (OWL) or Resource Description Framework (RDF) as the format for the knowledge description. RDF and OWL ensure suitable resources for the definition of information in a form comprehensible for software systems [ 10]. Before mentioning OWL and RDF, it is useful to review the use of Description Logic (DL). DL can be used to represent the robot’s information about its own world in formulated form [11].
Task planning is one of the main research topics in the robotic field. Xia et al. [ 12] pointed out the diversity and complexity of control parameters in task planning. They built ontologies for robot manipulators and analyzed the benefits of Ontology and Semantic Web Service for the robot control system using Web Ontology Language (OWL) and Resource Description Framework (RDF). According to their experimental results, the approach of the Semantic Web Service based on Ontologies increased the versatility of the system. In addition, it improved the real-time based application of remote robot control systems in industrial settings and enhanced the knowledge validity. Radakovic et al. [10] realized an intelligent manufacturing system using ontology languages such as OWL and RDF. Their study includes the recent application oriented aspects of reusable, resilient and modifiable agent based intelligent production control systems. They focused on their task planning capabilities within chosen packaging manufacture line scenario. Moreover, they aimed to combine information description techniques and deployment of ontologies as multi-agent system control in the industrial domain. In another study, AlMoadhen et al. [11] stated that the Semantic-Knowledge Based (SKB) plan generation for ambiguity in existing of objects. In addition, they proposed a new approach to build task plans using Markov Logic Networks (MLN) which is based on probabilistic values. They also applied test scenarios for a mobile robot using its OWL model.
RDF and OWL are commonly used languages which have information representation structures that can be appropriate to define robot environments via ontologies. In addition, they have eXtensible Markup Language (XML)-based file format to store formal definitions. OWL is a syntactical extension of RDF, a tool for representing knowledge semantically on the Semantic Web (SW) and providing an expressive and reasoning ability of Description Logic in the SW [10, 11, 13]. Knowledge in DL is composed of two key components. The first of these is referred to as terminological components (T-Box) which define ontology and specify the concepts, i.e. information that has no any changes, and the second one is called assertional components (A-Box) which define knowledge base and specify its instances, i.e. information may change according to their real world situation and continuously updated [10, 11, 13, 14]. Additionally, ontologies facilitate the automatic generation of axioms through a process known as reasoning [15].
Saeed et al. [13] presented a novel approach to using RDF in the robotics field. RDF is one of the fundamental semantic web tools for machine-readable data representation form of the semantic data model which is recommended by World Wide Web Consortium (W3C). An imperative aspect of this form is that it contains a subject, a predicate, and an object. XML is the language which has been used to describe RDF, which is an easy to understand and human readable form [16, 17, 18]. As part of their approach, they have simulated a mobile robot and its surrounding environment, as well as a camera sensor to assist the robot in navigating. They also used SPARQL protocol and RDF Query Language (SPARQL) in order to retrieve data from relevant sources. SPARQL protocol is designed to be used with RDF, and is similar to the Structured Query Language (SQL) structure. SPARQL is a well known query language and a protocol for collecting data [18]. A SPARQL query, which allows users to search for specific data in RDF, provides path finding and navigation for robots.
Eich et. al [14] have two scout robots and a mobile manipulator for space scenario and suggested an approach using spatial features and a fuzzy logic-based reasoning to identify objects and transfer the objects to a base station with ontology. In addition, they described T-Box and A-Box in their study. Yahya et. al [15] are also described T-Box and A-Box in their study. It is very important for intelligent manufacturing such as controlling knowledge related to resource optimization, equipment maintenance, and various production and services which are on-demand. They examined the usage of the Semantic Web and Knowledge Graphs for Industry 4.0 and recommended an improved Reference Generalized Ontological Model (RGOM) based on the Reference Architecture Model for Industry 4.0 (RAMI 4.0). Their goal is to highlight major problems and opportunities that could originate from the association of these existing technologies for Industry 4.0. Like Yahya et. al [15], Yılmaz et. al [16] also conducted a study in the Industry 4.0 domain. They created a semantic ontology using RDF to monitor diferent types of representative faults such as environmental, mechanical, and software errors in an Autonomous Mobile Robot (AMR). AMR is one of the significant components for the progress of the intelligent manufacturing field, and their usage in Industry 4.0 keeps enhancement. The Ontology API serves as the base for the knowledge layer in their study and they used SPARQL for querying the machine that supplies the order conditions. The ontology model is established with Jena Fuseki Server which is SPARQL endpoint server. A SPARQL query that runs the ordering criteria is realized based on the ontology. As a result, they stated that importance of RDF for Industry 4.0. Saeed et al. [17] used the RDF to define navigation of a mobile robot semantically in an ontology. They proposed a recent and fruitful way of expressing semantic relationships within the navigation ontology by using RDF helper nodes in real-time. In another study, Saeed et al. [18] proposed a system called Robot Semantic Protocol (RoboSemProc). RoboSemProc has provided the implementation of semantic technology from scratch for the process of semantic knowledge generation, real-time ontology population and natural language communication between human-robot interaction. They utilized Semantic Web Technology components which are OWL, RDF, SPARQL and Jena in their study. Wan et al. [19] suggested an ontology-based resource reconfiguration method for resource utilization. They used OWL, RDF, SPARQL and Jena components for their intelligent device ontology which defines the intelligent manufacturing resource. They created a smart manufacturing resource integration architecture based on their ontology. Later on, they analyzed the reconfiguration of the smart manipulator as an implementation case based on the ontology, and they verified its feasibility in intelligent manufacturing. Consequently, their study has provided a new technique for reconfiguration of producing resources.
Ji et al. [20] built a semantic knowledge ontology for representing two primary types of knowledge that are environmental description and robot primitive actions. Their method has combined the representation of semantic information with classical approaches in Artificial Intelligence (AI) to construct an elastic framework which can help service robots for task planning. Li et al. [21] aimed to assist robots to manipulate objects with the semantic limitations such as grasp location, grasp type, trajectory constraint which are learnt from human manipulation behaviors. They suggested a representation of human manipulation behaviors in machine comprehensible semantics and a collaborative reasoning concept and stated that the robot manipulation can be completed under “consciousness” and suitable for the object and task. Kunze et al. [22] presented Semantic Robot Description Language (SRDL) and inference mechanisms for describing robot components, capabilities and actions. SRDL uses OWL for modeling information and allows to perform inferences regarding the capability of the robots to make certain actions. Bouguerra et al. [23] described a novel intelligent execution monitoring approach for mobile robots using semantic knowledge in indoor environments. The main idea is to calculate expectations side by semantic domain knowledge, that can be monitored at execution time by the robot correctly. Thus, they presented a representation of the semantic information that is used to monitor execution for a real mobile robot.
In this section, we present our ontology-based approach to controlling robot trajectory based on semantic knowledge. Figure 1 illustrates the overview diagram of our proposed approach. The ontologies of the tasks and the environment are defined as specifications of the workplace in the main controller. Instead of being controlled by some GUI environment, the robot can be dynamically controlled by an ontology model. The main controller is also responsible for communicating with the simulator using a remote API. We employ an integrated robotic platform, which includes a robotic arm and hand, to perform a grasping task in the CoppeliaSim simulator. Control operations are performed according to the tasks contained in SPARQL queries. As a consequence, the robotic arm would perform tasks in response to those queries, and the queries would also enable revisions of the environment as per task completions. The following sections describe the proposed approach in more detail.
In this work, we focus on enabling the robotic arm to perform semantically enhanced tasks on a semantically described environment. We defined an autonomous robotic arm ontology (ARAO) that consists of the robot, task, and shape classes and their attributes as shown in Figure 2. The relationships between the classes are established. For instance, a robotic manipulator performs robotic tasks, and robotic tasks interact with shapes. ARAO presents how to model the basic robotic environment elements, and describes some basic robotic tasks and the relations between the tasks and the environment elements. The ontology defined with RDF (i.e. ARAO)) will be used for storing environmental information to monitor the robotic tasks. RDF description of one of the shape instances (i.e. a yellow cuboid) is shown in Figure 3.
In the ontology, two categories of robots are considered: mobile and non-mobile. In our study, a non-mobile robot that combines a robotic arm and hand is included. For task planning purposes, a description of robotic tasks is needed [24], and the ontology in this study also defines the robotic tasks. Table 1 summarizes the activities included in the hypothetical validation scenario that will be used in our simulated environment. These activities include robotic tasks that are defined in the ontology such that the robot can reach the boxes (i.e. an instance of the shape class) by arm and the hand can grasp or release the boxes [25, 26]. The shape class [27], which is composed of five data properties: coordinate, orientation, color, label and attribute, is essential for understanding and manipulating the robotic environment as depicted in Table 2.
In this study, the coordinate property (x, y, z) and the orientation property (ψ, θ, φ) values are only considered in the ontology that enable the robotic arm to locate the shapes and reach them with appropriate orientation. Attribute, color and label properties will be used to identify the sequence of tasks. The ontology defined in this study has a limited scope for now, but our research agenda includes expanding it in the near future.
Jena Ontology API will be used to retrieve data on RDF via SPARQL queries. SPARQL queries will retrieve and update environment information for the robotic manipulator performed in CoppeliaSim simulator. CoppeliaSim is a robotics simulation environment that is used for the development of robot systems, prototyping, and useful for validating algorithms. It is possible to establish a SPARQL-based querying integration with CoppeliaSim via Remote API properties. In this manner, a bidirectional communication can be achieved with the simulator and ontology. The Universal Robot UR5 robot arm and ROBOTIQ85 robot gripper is used in the simulation. The gripper is attached to the connection point of the robotic arm on the simulator and fixed to the ground. The table is placed in a convenient location so the robot arm could reach it. The height of the table from the ground is 60 cm. Green, blue and yellow boxes with side lengths of 5 cm, 4 cm, and 3 cm are placed on the table. The simulator environment created for executing a set of sequential activities that are composed of robotic tasks is shown in Figure 4 and the hypothetical validation scenario can be summarized as follows.
In the simulator environment, there is a yellow box at the top, a blue box in the middle, and a green box at the bottom. For each activity that intents to move a specific box, the task sequence is given as reach, grasp, reach and release, and the robot moves the boxes from source coordinates to target coordinates as shown in Table 1. The first activity is defined as “Move the yellow box”. After receiving the command of executing the activity, the source and target coordinates are taken from the ontology via a SPARQL query and send to CoppeliaSim via Remote API. Following this, the robot reaches and grasps the yellow box and then moves to the desired coordinate by using the inverse kinematics method. The new location of the yellow box is updated in the ontology after completing the activity with a release task by using a SPARQL query. After that, activity 2 is executed, which is ”Move the blue box” which refers to moving the blue box on top of the yellow box. Finally, activity 3 is executed that move the green box onto the blue box. The new locations of the shapes are updated in the ontology as per task descriptions and the robot changes its task plan based on the described order of the tasks.
In this study, we present a framework to investigate the relationships between robotic tasks and the existing objects in a robotic environment. For that purpose, an autonomous robotic arm ontology was defined to make the information about the environment explicit. This information will be stored in the ontology and then transferred to the robotic simulator using SPARQL queries via Jena Ontology API. The proposed ontology-based approach will enable the manipulator to perform basic robotic tasks in its environment. To validate and test our approach, we developed a hypothetical scenario that can be described as picking up and moving the boxes. Our approach can potentially control high-level task planning if the explicit information of the environments is available.
In future works, we plan to improve our ontology from the state of the art and by including more objects in the robotic environment. Thus, we will be able to extend the shape class in the ontology. We may even enhance the relationships between the task class and the shape class. In this way, we will be able to increase the number of manipulator components and add various tasks. While testing the approach with various other robotic manipulator scenarios, the ontology will extend and mature. Also, we plan to include some machine learning and image processing algorithms to recognize the objects. Thus, when new objects are added to the environment, these objects can be perceived by the robot and added to the ontology automatically, thereby improving the environment defined in the ontology. robot learning, Recent Global Research and Education: Technological Challenges 519 (2016) 349–355. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 3 1 9 - 4 6 4 9 0 - 9 _ 4 7 . [4] J. Yu, Y. Su, Y. Liao, The path planning of mobile robot by neural networks and hierarchical reinforcement learning, Front Neurorobot 14 (2020). doi:1 0 . 3 3 8 9 / f n b o t . 2 0 2 0 . 0 0 0 6 3 . [5] H. Masuta, N. Kubota, Environmental perception for a robot using 3d range camera, in:
World Automation Congress 2012, 2012, pp. 1–6. [6] N. Kubota, Visual perception for a partner robot based on computational intelligence, in: 2004 IEEE International Conference on Fuzzy Systems (IEEE Cat. No.04CH37542), volume 1, 2004, pp. 293–298. doi:1 0 . 1 1 0 9 / F U Z Z Y . 2 0 0 4 . 1 3 7 5 7 3 7 . [7] H. Masuta, N. Kubota, Perceptual system using spiking neural network for an intelligent robot, in: 2010 IEEE International Conference on Systems, Man and Cybernetics, 2010, pp. 3405–3412. doi:1 0 . 1 1 0 9 / I C S M C . 2 0 1 0 . 5 6 4 2 4 7 1 . [8] M. Cai, K. M. Kitani, Y. Sato, Understanding hand-object manipulation with grasp types and object attributes, in: Robotics: Science and Systems, 2016. [9] X. Zhang, J. Sinapov, S. Zhang, Planning multimodal exploratory actions for online robot attribute learning, 2021. doi:1 0 . 1 5 6 0 7 / R S S . 2 0 2 1 . X V I I . 0 0 5 . [10] M. Radakovič, P. Vrba, M. Obitko, Procedural and declarative knowledge in agent-based control, in: IECON 2010 - 36th Annual Conference on IEEE Industrial Electronics Society, 2010, pp. 1371–1378. doi:1 0 . 1 1 0 9 / I E C O N . 2 0 1 0 . 5 6 7 5 4 8 4 . [11] A. Al-Moadhen, M. Packianather, R. Setchi, R. Qiu, Automation in handling uncertainty in semantic-knowledge based robotic task-planning by using markov logic networks, Procedia Computer Science 35 (2014) 1023–1032. URL: https://www.sciencedirect. com/science/article/pii/S1877050914011533. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / j . p r o c s . 2 0 1 4 . 0 8 . 1 8 8 , knowledge-Based and Intelligent Information Engineering Systems 18th Annual Conference, KES-2014 Gdynia, Poland, September 2014 Proceedings. [12] H. Xia, Y. Liu, T. Song, B. Tan, Research on the industrial robot control system based on the ontology and web service, in: Proceedings of the 2007 International Conference on Intelligent Systems and Knowledge Engineering (ISKE 2007), Atlantis Press, 2007/10, pp. 713–718. URL: https://doi.org/10.2991/iske.2007.122. doi:h t t p s : / / d o i . o r g / 1 0 . 2 9 9 1 / i s k e . 2 0 0 7 . 1 2 2 . [13] N. T. M. Saeed, M. Fathi, K.-D. Kuhnert, An approach for instant conversion of sensory data of a simulated sensor of a mobile robot into semantic information, in: 2018 IEEE International Conference on Electro/Information Technology (EIT), 2018, pp. 0142–0146. doi:1 0 . 1 1 0 9 / E I T . 2 0 1 8 . 8 5 0 0 0 8 7 . [14] M. Eich, R. Hartanto, Fuzzy-dl perception for multi-robot systems, in: Proceedings of the 12th International Symposium on Artificial Intelligence, Robotics and Automation in Space (i-SAIRAS), ESA - European Space Agency, France, 2014, pp. 1–8. URL: https: //eprints.qut.edu.au/83991/. [15] M. Yahya, J. G. Breslin, M. I. Ali, Semantic web and knowledge graphs for industry 4.0, Applied Sciences 11 (2021). URL: https://www.mdpi.com/2076-3417/11/11/5110. doi:1 0 . 3 3 9 0 / a p p 1 1 1 1 5 1 1 0 . [16] M. Yilmaz, A. Yazici, E. Cinar, Rdf-based semantic for condition monitoring of autonomous mobile robot, in: 2021 2nd International Informatics and Software Engineering Conference (IISEC), 2021, pp. 1–5. doi:1 0 . 1 1 0 9 / I I S E C 5 4 2 3 0 . 2 0 2 1 . 9 6 7 2 3 4 1 . [17] N. T. Mohammed Saeed, C. Weber, M. Fathi, K.-D. Kuhnert, An eficient alternative for modeling spatial prepositions with rdf helper nodes based on the environment perception of a mobile robot, in: 2019 IEEE 28th International Symposium on Industrial Electronics (ISIE), 2019, pp. 1138–1143. doi:1 0 . 1 1 0 9 / I S I E . 2 0 1 9 . 8 7 8 1 4 2 1 . [18] N. Saeed, M. Fathi Kazerouni, M. Fathi, K.-D. Kuhnert, Robot semantic protocol (robosemproc) for semantic environment description and human–robot communication, International Journal of Social Robotics 12 (2020). doi:1 0 . 1 0 0 7 / s 1 2 3 6 9 - 0 1 9 - 0 0 5 8 0 - 5 . [19] J. Wan, B. Yin, D. Li, A. Celesti, F. Tao, Q. Hua, An ontology-based resource reconfiguration method for manufacturing cyber-physical systems, IEEE/ASME Transactions on Mechatronics 23 (2018) 2537–2546. doi:1 0 . 1 1 0 9 / T M E C H . 2 0 1 8 . 2 8 1 4 7 8 4 . [20] Z. Ji, R. Qiu, A. Noyvirt, A. Soroka, M. Packianather, R. Setchi, D. Li, S. Xu, Towards automated task planning for service robots using semantic knowledge representation, in: IEEE 10th International Conference on Industrial Informatics, 2012, pp. 1194–1201. doi:1 0 . 1 1 0 9 / I N D I N . 2 0 1 2 . 6 3 0 1 1 3 1 . [21] C. Li, G. Tian, Transferring the semantic constraints in human manipulation behaviors to robots, Applied Intelligence 50 (2020). doi:1 0 . 1 0 0 7 / s 1 0 4 8 9 - 0 1 9 - 0 1 5 8 0 - 8 . [22] L. Kunze, T. Roehm, M. Beetz, Towards semantic robot description languages, in: 2011 IEEE International Conference on Robotics and Automation, 2011, pp. 5589–5595. doi:1 0 . 1 1 0 9 / I C R A . 2 0 1 1 . 5 9 8 0 1 7 0 . [23] A. Bouguerra, L. Karlsson, A. Safiotti, Semantic knowledge-based execution monitoring for mobile robots, 2007, pp. 3693–3698. doi:1 0 . 1 1 0 9 / R O B O T . 2 0 0 7 . 3 6 4 0 4 4 . [24] Sun, Zhang, W. Chen, Rtpo: A domain knowledge base for robot task planning, Electronics 8 (2019) 1105. doi:1 0 . 3 3 9 0 / e l e c t r o n i c s 8 1 0 1 1 0 5 . [25] M. Diab, M. Gillani, A. Akbari, J. Rosell, An Ontology Framework for Physics-Based
Manipulation Planning, 2017. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 3 1 9 - 7 0 8 3 3 - 1 _ 3 7 . [26] B. Ozyer, E. Oztop, Task dependent human-like grasping, in: Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots, 2008, pp. 227–232. doi:1 0 . 1 1 0 9 / I C H R . 2 0 0 8 . 4 7 5 5 9 4 9 . [27] C. Ciorascu, C. Künzi, K. Stofel, Overview: Semantic management of 3d objects (2008).