Knowledge-based and particularly logic-based systems for task planning and execution guarantee trustability and safety of robotic systems interacting with humans. However, domain knowledge is usually incomplete. This paper proposes a novel framework for task knowledge refinement from realtime user feedback, based on inductive logic programming. The user feedback is used to generate cumulative examples which allow a learner to iteratively refine logic rules describing the task. Validated in a benchmark surgical training task, our system is also able to execute and learn unknown actions within reasonable time.
(a)
(b)
particular, we consider the training task of peg transfer (described in Section 2.1) from the
fundamentals of laparoscopic surgery [
ILP has been used by an international research community in diferent domains, particularly
to learn logic programs in AI [
The setup for peg transfer (Figure 1a) consists of 2 patient-side manipulators (PSMs) of dVRK equipped with graspers, and an external vision sensor (RGB-D camera). The grasping arms operate on up to four colored rings, with the goal to place them on the same-colored pegs. This task is subject to geometric constraints, e.g., each PSM can move only to reachable rings and pegs (depending on the relative distance among them). For instance, in Figure 1a, the red-dashed line defines regions of reachability for objects, e.g., the red ring and peg can be reached by PSM1, while PSM2 needs to pick the blue ring and transfer it to PSM1 before placement on the peg. Furthermore, rings may be initially placed on grey pegs, thus extraction may be required.
ASP [
:- closed_gripper(A, t), grasp(A, ring, C, t). state that a ring can be grasped if an arm is close to it (normal axiom) and that an arm with a closed gripper cannot grasp (constraint), respectively. t is a variable for temporal planning, representing a discrete time step. Symbol :- is ASP syntax for logic implication ← .
This section provides salient concepts from the Inductive Learning of Answer Set Program
(ILASP), the state-of-the-art tool for learning axioms explaining answer sets [
A generic ILP problem under the ASP semantics is defined as the tuple = ⟨, , ⟩, where is the background knowledge, i.e. a set of axioms, atoms and variables in ASP syntax; is the search space, i.e. the set of candidate ASP axioms that can be learned; finally, = + ∪ − is a set of examples. The goal of is to find a subset ⊆ such that ∪ |= . Examples are Context-Dependent Partial Interpretations (CDPIs), i.e., couples ⟨, ⟩, being a partial interpretation and the context. A partial interpretation is a set of ground atoms, corresponding to actions for the purpose of this paper. Similarly, the context is a set of ground atoms representing environmental variables. Examples may be positive (∈ +), i.e., partial interpretations possibly generated solving ∪ ∪ , or negative (∈ − ), i.e., partial interpretations never generated solving ∪ ∪ . This means that positive examples typically induce normal axioms (e.g., preconditions and efects of actions), while negative examples generate constraints (cautious induction).
ILASP finds the minimal hypothesis, i.e., with the least number of axioms and atoms, covering examples.
We propose a framework for refining (surgical) robotic task knowledge during execution, thanks
to continuous feedback by human experts supervising the autonomous behavior (see Figure 1b).
The basic workflow is the following:
1. a task/domain knowledge base is initialized with available prior information (e.g., main
task resources and specifications) in ASP formalism; for complex tasks, prior information
can be extracted from domain books [
(a) Grasping. (A) (b) Extraction. (U,L).
(c) Go to peg. (U,L) (d) End of task. (A) 2. an autonomous planning/execution module, e.g., the one proposed in [20, 21, 22], interprets sensor information in order to ground environmental atoms (context ) in ASP, and reason on task knowledge to generate a sequence of actions. This module is also in charge of motion planning and execution according to each specific action; 3. before executing each action planned by the reasoner, feedback is asked to an expert user, in order to preserve safety. The user may either confirm or reject the proposed action. The user can also propose a specific action to be executed, in case the autonomous planner is not able to find one in the current context; 4. the human feedback, the action, and the context from the planning/execution module, generate a CDPI stored in a list of examples. Specifically, each accepted/proposed action corresponds to is saved in +, while rejected actions are in − . For computational convenience, we store examples for each action separately; 5. when a negative example is generated, say for action a, an ILASP-based learning module considers all examples for a and , except for axioms including a. In fact, these axioms have generated a wrong plan according to the expert user, so they are probably incorrect. Hence, new axioms are generated for a and added to . Discarding axioms for a does not afect the computational burden of learning since ILASP complexity depends on the size of , hence the number of context atoms.
We perform a preliminary validation of our framework in a simulated environment in CoppeliaSim2 replicating the dVRK setup for peg transfer and autonomous planning/execution framework with Clingo [23] for ASP solving (plan computation) as in [20]. The human feedback is given to the system via the command line, with the user typing either yes/no for confirmation of every single action, or a specific action to be executed as a predicate in ASP syntax. The goal is to refine axioms defining preconditions of actions and constraints.
We assume that the autonomous system is not aware of extract(A, ring, C) action, needed when a ring is on a peg before moving it away. We consider the initial scenario in Figure
2a. After grasping, the autonomous planner proposes move(psm2, peg, yellow). However, the action is infeasible, so the user forbids it and commands extract(psm2, ring, yellow). At this moment, the list of examples contains only a positive example for extraction and one negative for moving to peg. Corresponding ASP axioms are removed from the task knowledge base, and ILASP runs two parallel learning tasks, leading to an empty hypothesis for moving to peg (no positive examples) and the following axiom as a precondition for extraction: extract(A, ring, C, t) :- in_hand(A, ring, C, t).
After extraction (Figure 2b), the autonomous system should plan move(psm2, peg, yellow), but axioms for this action have been removed at the previous learning stage. Hence, the user is required to input the action to be executed (Figure 2c), introducing a new positive example for it. Hence, the following axioms can be learned: move(A, peg, C, t) :- reachable(A, peg, C, t).
:- in_hand(A, ring, C, t), on(ring, C, peg, C2, t), move(A, peg, C3, t). with the latter constraints specifying that moving to the peg is not possible when the ring is on another peg. Finally, in Figure 2d the autonomous system can complete the task for the blue ring (executing extraction correctly) with the newly learned axioms. A video of the execution can be found at https://bit.ly/AIRO2022. Overall, the learner is invoked 3 times, with a maximum learning time of ≈ 6 s. For each action, includes ≈ 400 axioms.
We have presented a system based on user’s feedback to refine existing task knowledge, in the form of a logic program (e.g., an ASP program). Preliminary results in a paradigmatic training task for robotic surgery show that our system is able to propose actions to an expert, execute them only in case of approval (supervised autonomy for safety), accept commands of possibly unknown actions from the expert and exploit them to iteratively refine task knowledge for future re-use in autonomous execution.
Future research will focus on the improvement of the existing system, in order to adopt it in a real robotic setup. In particular, learning of motion primitives associated with unknown actions shown by the user will be needed, e.g., with dynamic movement primitives [24]. Moreover, in order to increase the usability of the framework in a real surgical setup, the user should be able to provide feedback via vocal commands. For instance, with reference to Figure 2a the user could state extract the yellow ring, then move it to the yellow peg, which could then be interpreted with natural language processing based on ad-hoc linguistic resources (e.g., [18]), and translated to ASP syntax for knowledge refinement. Finally, we remark that learning times with ILASP can be further improved with, e.g., ad-hoc incremental solvers [25].
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 742671 “ARS”). s11548-021-02370-9. [18] M. Bombieri, M. Rospocher, S. P. Ponzetto, P. Fiorini, The robotic surgery procedural framebank, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 3950–3959.
URL: https://aclanthology.org/2022.lrec-1.420. [19] M. Bombieri, M. Rospocher, S. P. Ponzetto, P. Fiorini, Machine understanding surgical actions from intervention procedure textbooks, Comput. Biol. Medicine 152 (2023) 106415. URL: https://doi.org/10.1016/j.compbiomed.2022.106415. doi:10.1016/j.compbiomed. 2022.106415. [20] M. Ginesi, D. Meli, A. Roberti, N. Sansonetto, P. Fiorini, Autonomous task planning and situation awareness in robotic surgery, in: IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2020, pp. 3144–3150. [21] D. Meli, et al., Autonomous tissue retraction with a biomechanically informed logic based framework, in: 2021 IEEE Int. Symp. on Medical Robotics (ISMR) [in press], IEEE, 2021, pp. 0–7. [22] E. Tagliabue, D. Meli, D. Dall’alba, P. Fiorini, Deliberation in autonomous robotic surgery: a framework for handling anatomical uncertainty, in: Proceedings - IEEE International Conference on Robotics and Automation, 2022, pp. 11080–11086. [23] M. Gebser, R. Kaminski, B. Kaufmann, M. Ostrowski, T. Schaub, P. Wanko, Theory solving made easy with clingo 5, in: Technical Communications of the 32nd International Conference on Logic Programming (ICLP 2016), Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2016, pp. 2:1–2:15. [24] M. Ginesi, D. Meli, A. Roberti, N. Sansonetto, P. Fiorini, Dynamic movement primitives: Volumetric obstacle avoidance using dynamic potential functions, Journal of Intelligent & Robotic Systems 101 (2021) 1–20. [25] M. Law, K. Broda, A. Russo, Search space expansion for eficient incremental inductive logic programming from streamed data, International Joint Conference on Artificial Intelligence (2022).