This study explores gesture interpretation by utilizing an ontological context. The aim is to store gestures and scene context data in an ontology and use its knowledge graph to actuate the robot arm to perform sets of manipulation tasks used in various environments. The knowledge graph captures the relationships between gestures, objects in the scene, and the desired actions. By putting the ontological context into use, the system can understand the meaning behind the gestures and execute the appropriate actions. The paper focuses on the development of the ontology, including the creation of class properties and the embedding of gestures within the ontology. Additionally, the paper explores how the integration of specifying context interpretation from the ontology may look to enhance the interpretation of gestures. The proposed approach aims to provide more intuitive and adaptive gesture-based supervisory control of robots in general. We tested the proposed ontological system in several tests so that it may be used in our future applications.
Human-robot interaction (HRI) plays a crucial role in enabling robots to assist humans in various tasks. One important aspect of HRI is the ability of robots to understand and interpret human gestures accurately. Gesture recognition allows robots to perceive and respond to human commands and intentions, enhancing their usability and effectiveness in assisting humans. Similarly, the ability to understand and represent gestures enables robots or virtual agents to behave in a way that is easily interpretable by humans.
Robot control using hand gestures so far is mainly considered a direct mapping between
gestures and actions without any context of the environment (e.g. [
In this work, we link these two worlds by proposing a robust knowledge representation for the
interpretation of gestures within the context of realistic robotic environments. To achieve accurate
and context-aware gesture recognition, we built upon our previous work [
The kitchen environment presents unique challenges for robots, as it is relatable and understandable for humans and contains various basic manipulation tasks such as food preparation, desk organization, and cleaning in a kitchen setting. In these scenarios, a subset of objects from the YCB dataset [6] is utilized consisting of kitchen equipment and food items.
The proposed ontology and related code is available online1.
First, we review related work about the usage of human gestures for robot control. Second, we summarize the most significant works on modeling and representing gestures.
In previous works, hand movements were used to teleoperate the robot by directly mapping
the user’s hand to the robot end-effector [7]. In [8], hand motions recognized by a combination
of depth cameras and inertial measurement units (IMUs) were used for robot teaching. Another
way of control is commanding gestures using action signaling (e.g., [
Most of the current gesture-controlled devices (e.g., Hi5 VR Glove [9]) were used only experimentally. Only a few (e.g., Leap Motion [10], Oculus Quest’s hand tracking) made it into mass production and even fewer people started using them on a daily basis. One example is the smartphone industry, where developers used hand gestures to control basic controls, e.g., music [11] or resizing pictures. Gestures are also used in virtual reality, including environment control [12]. In contrast to these approaches, all consider only a very simple notion of context. Our hypothesis is that using situation context from a rich knowledge graph is the key to making gestures a reliable and natural means of communication.
The modeling of gestures is in many cases rather informal (e.g., [13]), or restricted to geometrical characteristics (e.g., [14], [15]). For example, Ousmer et al. [14] decomposes gestures into different segments with associated hand poses to support the recognition of gestures. In contrast, our goal is rather to support their interpretation, and thus geometrical characteristics are not sufficient. A well-formalized account is provided by the SUMO ontology which focuses on modeling the communication underlying a gesture [16]. However, gestures are not characterized by the affordances of the environment in the SUMO ontology. Another related ontology is the 1https://github.com/petrvancjr/gesture-ontological-context-interpreter (a) Bowl (b) Cheez (c) Foam (d) Mug
It (e) Mus- (f) SPAM (g) Sugar (h) Tomatard toes
HDGI ontology [17]. It is designed for human-device interaction and puts particular emphasis on the link between gestures and device context and affordances. However, here we consider human-robot interaction which requires a more detailed representation of affordances and context.
In most works the scene context is not taken into account, e.g. the system uses fixed mapping
from gestures to robot task [
In this section, we will describe the considered application domain including the utilized objects (Sec. 3.1), robot actions (Sec. 3.2), and gestures (Sec.3.3). For the gesture detector, we use the Leap Motion Controller [10] and Franka Emika Panda [19] as our robot which is popular to be used for human-robot interaction while being able to easily manipulate the objects.
The scenario in consideration comprises a table and several items that are on top of it. The objects are a subset of the popular YCB dataset [6] because they are objects which people interact with on a daily basis. We took mainly kitchen equipment and food items to accomplish specified tasks, which we will describe in the next section. The objects are randomly arranged on the scene within every new task. See Fig. 1 for the set of selected objects.
We defined the number of actions the robot can perform while still being able to accomplish given scenarios. The final list consists of 7 actions: Pick, Pour, Put, Place, Move-up, Move-right, and
Move-left. Actions Pick, Pour, Put, and Place are tied to a specified object. Action Place places the object into specified storage, the storage defined within reach on the side of the table.
Action Move-up moves the robot end-effector into the home (upper) position. Actions Moveleft or Move-right move the robot end-effector according to the common coordinate axis by a certain amount. Additional object-focused features might be added for convenience. For example, if an object is near the end-effector position, the position is adapted to attain the position above the given detected object.
Additionally, the system supports a non-robotic action Select-object which computes probabilities of objects being selected based on the direction of the pointing finger (see Sec. 3.3 for more details).
Our system utilizes two gesture types: action gestures and point (deictic) gestures.
The first action gesture type is defined based on gestures taxonomy [ 20], the Semaphoric Gestures Model. It is used to discretely classify a pre-trained set of gestures. We selected a set of 8 action gestures that are shown in Fig. 2. The detectors used in our system are returning the confidence of each gesture from this set in real-time. When any gesture has enough evidence, its data are written into the ontology. The gesture detection uses a combination of static and dynamic gesture detectors. The static detector uses a single time-frame hand structure (see Fig. 4), from which then the feature vector (of length 57) is extracted for gesture classification. On the other side dynamic gesture detector uses a moving time frame of hand movement but only hand pose as the feature is used. The detectors are combined to form the final Compound gesture, resp. specific hand configuration plus movement (see Fig. 2 description).
Point (deictic) gestures have the effect of triggering the procedure of object choice. The procedure chooses the closest object to the user’s pointed line. This involves the calibration with a scene [21]. The poses of objects are retrieved from the knowledge graph which saves recent object positions. We use the CosyPose detection method [22] to get 6DoF poses of objects.
The ontological nature of gestures appears to be somewhat diverse. Gestures are, on the one hand, acts of non-verbal communication where an agent attempts to convey some information Disposition to do tasks Picking Pouring ...
Sender
has Human component
Arm is performed by Receiver
Communication
Task Message
Gesture Action executes task
Object Patient Agent ... constrained by
Disposition set
BeingPickable ... has quality to other agents in its surrounding. They are, on the other hand, also bodily expressions in the form of postures and motions. This observation is captured, for example, in the SUMO ontology where a gesture is seen as any body motion which is also an instance of communication [16]. Another foundational account for an ontological notion of gestures is included in the ontology for Information Objects where a gesture is seen as an instance of bodily motion, and each bodily motion is seen as the realization of an abstract piece of information[23]. However, such foundational definitions are rather vague about the actual meaning of gestures. In our implementation, we commit to the IO ontology definition simply for pragmatic reasons as the alignment with the adopted knowledge framework was less cumbersome.
The intended meaning of a gesture can often only be understood when taking context into account. A hand gesture indicating a stop signal, for instance, is vague and can only be understood if it is clear to which objects and actions it refers. Another aspect is that the detection of gestures could be wrong, but that the falsely detected gesture does not make sense in the current context in which case it could be discarded or re-classified. This suggests the importance of relationships between gestures and the context in which they occur. These relationships are of primary concern for us, but they are rarely considered in related literature about gesture ontologies or are not designed for robotics use cases.
Nevertheless, it is worth investigating to what extent existing ontologies that link gestures and context could be adopted for robotics use-cases. To this end, we adopt the HDGI ontology [17] as it defines a rather comprehensive model of gestures that also includes links to device affordances and context. The ontology does however not model interactions between agents and what is afforded to them. But this interaction is important for robotics use-cases as robots have different capabilities to execute an action. Thus, we rather employ a more fine-grained notion of affordance from an existing ontology that was designed with robotics use-cases in mind [24]. We consider an affordance as the description of a disposition and a disposition is an absolute property, it does not depend on a context. Furthermore, the ontology defines affordances as the descriptive context between dispositional pairs and thus can express a relation between two disposed objects. For example, the robot is disposed to handle small-sized objects while a small-sized object is disposed to be grasped, carried, thrown, and so on. The concepts and relations used or defined in the proposed ontology are shown in Fig. 3.
The central notion is the Gesture concept. It is characterized as a type of task, i.e. as a sub-concept of CommunicationTask. The notion of task is imported via the robotics affordance ontology [24] which in turn imports the foundational ontology DOLCE+DNS Ultralite (DUL) [25]. Following DUL, tasks describe how certain events are to be interpreted, executed, etc. In the case of CommunicationTask, this is done through the three roles linking sender, receiver, and message to the task. For gestures, the role of the message is actually taken by the event that executes the gesture, i.e. the act of performing the gesture is the message transported. Finally, we say that each gesture is defined in one or more affordances representing some action potential in the environment. For example, gesture grab has affordance representing picking the object. Definition 1. A gesture is a communication task that is defined in an affordance.
An affordance further defines a task related to affordance and creates constraints for objects taking the roles of the task. Namely, those objects need to be the host of certain dispositions. For example, an affordance of picking up an object may refer to a task where the picked object must be a host of the pickable disposition, and where the agent must be the host of the can-pick capability (disposition).
The gesture concept is further decomposed into three cases: DeicticGesture, ActionGesture, and ComposedGesture. First, a deictic gesture is a pointing gesture used to draw the attention of the receiver to a particular object or region of interest. Second, an action gesture refers to a task request. Finally, a composed gesture is a combination of several gestures during an episode. An instance of ComposedGesture is created each time the gesture episode ends. In our implementation, this depends on our hand sensor [10], which has a limited field of detection. The end of the episode is defined as the hand disappearing from the detection area. Each gesture event gives us data about confidence, timestamp, name of gesture which has been triggered, and potentially the relevant selected object, on which the user wants to work.
Features of scene objects are defined as object properties. These properties are shown in Figure 5 with relation to object type (e.g., Bowl). Properties are hand-picked based on our needs: Sizes (SmallSize and LargeSize) based on the ability to fit inside the PandaGripper, Color as a visual property, Sharpness property defining rounded or sharp objects (this property might be used when clarifying object choice), and finally Object types defining whether the object is a container and of which type (Liquid-container, or Object-container). Dynamic properties are supposed to be updated in real-time to keep the world representation up-to-date. They include Human
RightArm Arm
LeftArm Hand
Finger Elbow Pose Bone has property has component current Container capacity, Accessibility (disposition to be interacted, e.g. object is on top of a stack or in reach by robot’s gripper), or if an object has been already manipulated, which can help estimate the user’s next choice based on their preference. Geometric parameters include instant Pose value. Perception methods might be used for some properties, e.g. get pose (in our case by CosyPose method [22]) of the object. If no such technique is available, the logic methods might be used, e.g. when action Pour is done, the capacity of a container changes.
In this chapter, we discuss the potential approach for acquiring experience in the context of gesture interpretation using ontological context. While this work is currently in progress, we outline the steps and considerations that could be taken to acquire the necessary to test the usage of our knowledge graph and the possibility to improve the context-based gesture control system.
To acquire experience in gesture interpretation, a large dataset of human-robot interactions needs to be collected. This dataset should include our defined kitchen scenarios and tasks in which humans interact with the robot using described gestures. From the interactions, the data about the user’s Task instance reference for the current context (Dynamic properties of objects and applied gesture). The next step is to take advantage of ontologies and scale the number of object properties.
Instantaneous object properties are written in real-time into ontology. As we discussed poses of objects [22] and other properties experimentally or by hand, e.g. current capacity of Liquidcontainer. Accessibility of an object by the property, that the given object is in a predefined boundary, and checker to estimate if no other object is detected on top of the given object.
The Arm instances also enable us to store raw hand movements. Based on this data structure, we may run gesture set classification that uses collected data training and improving discussed gesture recognizers.
Representing world and gesture information as ontology enables us to define the knowledge of the world in the right format with the possibility to scale its properties by size without getting confusing. One of the ways is to generate a better embedding vector for context-dependent action generation.
The following steps involve the validation of the proposed setup and a comparison test of how useful is using the knowledge graph to other methods in terms of scaling, for example, the number of properties on context-based action estimation. Last but not least, it would be interesting to explore the assembly of context vector embeddings with automatic methods. This would involve some scraping method, evaluator, and discriminator, to evaluate if the context vector is chosen properly for a given environment.
Petr Vanc was supported by CTU Student Grant Agency (reg. no. SGS23/138/OHK3-027/23). Karla Stepanova was supported by the Czech Science Foundation (project no. GA21-31000S). Daniel Beßler was supported by the German Research Foundation DFG, as part of Collaborative Research Center (Sonderforschungsbereich) 1320 Project-ID 329551904 “EASE - Everyday Activity Science and Engineering”, University of Bremen (http://www.ease-crc.org/). [5] P. Vanc, K. Stepanova, J. K. Behrens, Controlling robotic manipulations via bimanual gesture sequences, in: 2022 IEEE International Conference on Robotics and Automation (ICRA) Workshop, 2022, p. 2. [6] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, A. M. Dollar, The YCB object and model set: Towards common benchmarks for manipulation research, in: 2015 International Conference on Advanced Robotics (ICAR), 2015, p. 510–517. doi:10.1109/ICAR.2015. 7251504. [7] W. Zhang, H. Cheng, L. Zhao, L. Hao, M. Tao, C. Xiang, A gesture-based teleoperation system for compliant robot motion, Applied Sciences 9 (2019) 5290. [8] G. Du, M. Chen, C. Liu, B. Zhang, P. Zhang, Online robot teaching with natural human–robot interaction, IEEE Transactions on Industrial Electronics 65 (2018) 9571–9581. doi:10.1109/TIE.2018.2823667. [9] B. R. Glowacki, R. Freire, An open source etextile vr glove for real- time manipulation of molecular simulations, 2019. arXiv:1901.03532. [10] F. Weichert, D. Bachmann, B. Rudak, D. Fisseler, Analysis of the accuracy and robustness of the leap motion controller, Sensors 13 (2013) 6380–6393. doi:10.3390/s130506380. [11] S. Mccarthy, LG touch-less interface smartphone will unveil at MWC 2019 in Barcelona,
UWIRE Text (2019) 1–1. [12] C. Khundam, First person movement control with palm normal and hand gesture interaction in virtual reality, in: 2015 12th International Joint Conference on Computer Science and Software Engineering (JCSSE), 2015, pp. 325–330. doi:10.1109/JCSSE.2015.7219818. [13] E. M. N. Arendttorp, K. Rodil, H. Winschiers-Theophilus, C. Magoath, Overcoming legacy bias: Re-designing gesture interactions in virtual reality with a san community in namibia, in: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 1–18. URL: https://doi.org/10.1145/3491102.3517549. doi:10.1145/3491102.3517549. [14] M. Ousmer, J. Vanderdonckt, S. Buraga, An ontology for reasoning on body-based gestures, in: Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS ’19, Association for Computing Machinery, New York, NY, USA, 2019.
URL: https://doi.org/10.1145/3319499.3328238. doi:10.1145/3319499.3328238. [15] W. Khairunizam, K. Ikram, S. A. Bakar, Z. M. Razlan, I. Zunaidi, Ontological framework of arm gesture information for the human upper body, in: M. H. A. Hassan (Ed.), Intelligent Manufacturing & Mechatronics, Springer Singapore, Singapore, 2018, pp. 507–515. [16] A. Pease, I. Niles, J. Li, The suggested upper merged ontology: A large ontology for the semantic web and its applications, in: Working notes of the AAAI-2002 workshop on ontologies and the semantic web, volume 28, 2002, pp. 7–10. [17] M. Perera, A. Haller, S. J. Rodríguez Méndez, M. Adcock, HDGI: A human device gesture interaction ontology for the internet of things, in: The Semantic Web – ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2–6, 2020, Proceedings, Part II, Springer-Verlag, Berlin, Heidelberg, 2020, p. 111–126. URL: https://doi.org/10.1007/978-3-030-62466-8_8. doi:10.1007/978-3-030-62466-8_8. [18] S. Jain, B. Argall, Recursive bayesian human intent recognition in shared-control robotics, in: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018, p. 3905–3912. doi:10.1109/IROS.2018.8593766. [19] Franka Emika GmbH, Panda: The versatile industrial robot arm, 2018. URL: https://www.
franka.de/. [20] M. Karam, m. c. schraefel, A Taxonomy of Gestures in Human Computer Interactions,
Project Report, University of Southampton, 2005. URL: https://eprints.soton.ac.uk/261149/. [21] P. Vanc, J. K. Behrens, K. Stepanova, V. Hlavac, Communicating human intent to a robotic companion by multi-type gesture sentences, in: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), to appear in, 2023. arXiv:2303.04451. [22] Y. Labbé, J. Carpentier, M. Aubry, J. Sivic, Cosypose: Consistent multi-view multi-object 6d pose estimation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2020. [23] E. M. Sanfilippo, L. Jeanson, F. Laroche, Towards an ontology for information objects, 2018. URL: https://hal.science/hal-02498472, 1st workshop on semantic web technologies for human and social sciences, SWTHS, Catania, Italie. [24] D. Beßler, R. Porzel, P. Mihai, M. Beetz, R. Malaka, J. Bateman, A formal model of affordances for flexible robotic task execution, in: Proc. of the 24th European Conference on Artificial Intelligence (ECAI), 2020. URL: https://ai.uni-bremen.de/papers/ bessler20affordances.pdf. [25] A. Gangemi, The DOLCE+DnS Ultralite ontology (DUL) Version 4.0, 2021. URL: http: //www.ontologydesignpatterns.org/ont/dul/DUL.owl.