Interacting with unknown objects, and learning and producing e↵ective grasping procedures in particular, are challenging problems for robots. This paper proposes an intrinsically motivated reinforcement learning mechanism for learning to grasp uknown objects. The mechanism uses frustration to determine when grasping of an object is not possible. The critical threshold of frustration is dynamically regulated by impulsiveness of the robot. Here, the artificial emotions regulate the learning rate according to the current task and performance of the robot. The proposed mechanism is tested in a real world scenario where the robot, using the grasp pairs generated in simulation, has to learn which objects are graspable. The results shows that the robot equipped with frustration and impulsiveness learns faster than the robot with standard action selection strategies providing some evidence that the use of artificial emotions can improve the learning time.
Robots need e↵ective grasp procedures to interact with and manipulate unknown objects. In unstructured environments, challenges arise mainly due to uncertainties in sensing and control, and lack of prior knowledge and model of objects. E↵ective learning methods are essential to deal with these challenges. One classic approach here is to use reinforcement learning (RL) where an agent actively interacts with an environment and learns from the consequences of its actions, rather than from being explicitly taught. An agent selects its actions on basis of its past experiences (exploitation) and also by new choices (exploration). The goal of an agent is to maximize the global reward, therefore the agent needs to rely on actions that led to high rewards in the past. However, if the agent is too greedy and neglects exploration, it might never find the optimal strategy for the task. Hence, to find the best ways to perform an action they need to find a balance between exploitation of current knowledge and exploration to discover new knowledge that might lead to better performance in the future.
We propose a competence-based approach to reinforcement learning where exploration and exploitation is balanced while learning to grasp novel objects. In our approach, the dynamics of balancing between exploration and exploitation is tightly related to the level of frustration. The failures in obtaining a new goal may significantly increase the robot’s level of frustration, and push it into searching new solutions in order to achieve its goal. However, a prolonged state of frustration, when no solution can been found, will lead to a state of learned helplessness, and the goal will be marked as unachievable at the current state (i.e., object not graspable). Simply speaking, an optimal level of frustration favours more explorative behaviour, whereas low or high level of frustration favours more exploitative behaviour. Additionally, we dynamically change the robot’s impulsiveness that influences how fast the robot gets frustrated, and indirectly how much time it devotes to learning a particular task.
To demonstrate the advantages of our approach, we compare it with three other action selection methods: "-greedy algorithm, softmax function with constant temperature parameter, softmax function with variable temperature depending on agent’s overall frustration level. The results shows that the robot equipped with frustration and impulsiveness learns faster than the robot with standard action selection strategies providing some evidence that the use of artificial emotions can improve the learning time.
The rest of the paper is organized as follows. We first present related work in the area. Then, we give the details of the learning system including visual processing of objects, the RL framework and the proposed action selection strategies. In the next section, we present the experimental results and then conclude the paper. 2
Our main focus is on learning graspability of objects. Previously, analytical
methods are proposed for grasping objects [
In our work, we use reinforcement learning (RL) framework for learning and
incorporate competence-based intrinsic motivation for guidance in search. The
complexity of reinforcement learning is high in terms of the number of
stateaction pairs and the computations needed to determine utility values [
In our work, we use a competence-based approach for intrinsic motivation
for balancing exploration in RL. Frustration level of the robot is taken into
account. We further extend this approach by adopting an adaptive frustration
level depending on a task. Intrinsic motivation is investigated in earlier works.
Lenat [
We propose an intrinsically motivated reinforcement learning system for robots to learn graspability of unknown objects. The system includes two main phases for determination of grasp points on objects and experimentation of them in the real world (Fig. 1). The first phase includes the required methods to determine candidate grasp point pairs in simulation. Note that a robot arm with a twofingered end e↵ector is selected as the target platform. For this reason, grasp points are determined as point pairs. In the second phase of the system, the grasp points determined in the first phase are experimented in the real world through reinforcement learning. The following subsections explain the details of these processes. 3.1
In our system, objects are detected in the scene by using an ASUS Xtion Pro
Live RGB-D camera mounted on a linear platform for interpreting the scene for
tabletop manipulation scenarios by a robotic arm. We use a scene interpretation
system that can both recognize known objects and detect unknown objects in
the scene [
Transfer Point Cloud
to the Simulation Environment The Simulation Environment Transfer Candidates
to the
Real World Frustration Based Decision About Graspability
Real World Experimentation via
Reinforcement
Learning
The Real World Environment with the Robot and Objects
and finds the outlier 3D edges by RANdom SAmple Consensus (RANSAC)
algorithm [
Detection of Candidate Grasp Points in the Simulator Objects are represented by their center of masses (µ) and 3D edges (H). Then candidate grasp point pairs (⇢ =[p1, p2]) are determined as in Algorithm 1. In the algorithm, initially the reference points are determined. The center of mass, the upside and the bottom side center points are chosen as references. Based on these points, cross section points coplanar with the reference points and parallel to the table surface are determined. In the next step, the algorithm detects the closest point to the reference points on the same planar and draw a line crossing with reference points and closest to it. The second step is determining the opposite point to the closest one on the same line. This procedure continues until all points are tested. The algorithm produces the candidate grasp pairs (two grasp points with x,y,z values) and orientation of each pair according to (0,0) point in 2D (x,y) plane. These grasp points are tested in the simulator for finding out only the feasible ones.
In Fig. 2, the edges and sample grasp points for six di↵erent objects along with the number of grasp points are presented. Algorithm 1 Grasp Point Detection (µ, H)
Input: Object Center Of Mass µ, Edge Point Cloud H Output: Grasp Pairs P Detect maxZ, minZ and C as reference point ref . for each reference point do cP oints = findPointsOnTheSamePlane() mP oint = findClosestPointToReferencePoint(cP oints) slope =findSlope(mP oint,ref ) for each p 2 cP oints do
P slope =findSlope(mP oint,p) if onTheSameLine(P slope,slope) then
P { p, mP oint } end if end for end for
Edge Detection
Edge Detection
Edge Detection Simulation Representation
Simulation Representation
Simulation Representation 84 Grasp points are detected 48 Grasp points are detected 56 Grasp points are detected Edge Detection
Edge Detection
Edge Detection Simulation Representation
Simulation Representation
Simulation Representation 46 Grasp points are detected 92 Grasp points are detected 116 Grasp points are detected In the system, the output of the simulation environment is fed to the robotic arm to apply real-world experimentation. Intrinsic motivation with frustration level and new proposed impulsiveness method are evaluated to increase the learning process speed for the robot in order to give up quickly for the objects that are not graspable.
The main task of the robot is to learn which objects are graspable. We use
a Reinforcement Learning (RL) framework with Q-learning [
Ov] where, µ is the center of mass of the object, ⇢ is the selected set of two grasp points ⇢ =[p1, p2], is the grasp orientation, ! is the approach direction of the gripper and Ov is the 3D translation vector for object during grasp trial. A collision between the robotic arm and the object may occur when there is a trajectory error that results in a non-zero vector.
Actions can be represented as follows, where, ||Rv|| is the slide amount on the x axis and ! represents the approach vector to the object of interest.
In our framework, the robot receives the reward value of 10 (Rmax) when
the grasp is successful and 0.1 (Rmin) when the grasp is unsuccessful [
Q0(s, a) = Q(s, a) + ↵ ⇤ [R + ( ⇤ maxQ(s0, a)) Q(s, a)] (3) where, Q0(s, a) the next Q-value for state action pair (s, a), Q0(s, a) is the current Q-value, ↵ is the learning rate, R is the immediate reward after performing an action a in state s, is the discount factor, maxQ(s0, a) is the maximum estimate of optimal future value.
We investigate four action selection strategies. The first (and the simplest) one is the "-greedy action selection method (M1). This method most of the time selects the action with the highest estimated action value, but once in a while (with a small probability "), selects an action at random, uniformly, independently of the action-value estimates.
The second one is the SoftMax Action selection (Eq. 4) method (M2)with
constant temperature value [
A = [||Rv||, ! ] P (a)t =
eQt(a)/⌧ Pn b=1 eQt(b)/⌧ (1) (2) (4) where, P (a)t is the probability of selecting an action a at the time step t, Qt(a) is the value function for an action a, and ⌧ is the positive parameter called the temperature that controls the stochasticity of a decision. A high value of the temperature will cause the actions to be almost equiprobable and a low value will cause a greater di↵erence in selection probability for actions that di↵er in their value estimates.
The third strategy (M3) also uses the Softmax action selection rule. In this
approach, however, the ⌧ parameter is flexible and changes dynamically in
relation to the robot’s level of frustration and sense of control [
Z + (T t) where expectancy represents the probability of getting the highest estimated action value (as in the greedy action selection method), value refers to the expected action reward (here value = Rmax), Z is a constant derived from when rewards are immediate, indicates agent’s sensitivity to delay (impulsiveness) and (T t) refers to the delay of the reward in terms of “time reward” minus “time now”.
The impulsiveness is main focus of ours to develop interaction with frustration rate competence based motivation. According to triad ”Frustration Impulse - Temper”, a person who has high impulsiveness is considered as ”short tempered” and it means quickly get frustrated so that changes on frustration level for learning behavior. Our proposal with that, di↵erent values on impulsiveness directly a↵ect rate of leak, L, on frustration formula so frustration rate of agent also will be dependent on impulsiveness.
The robot apart from learning how to grasp an object, also needs to learn whether the target object is graspable or not. The learning of a selected grasp pair ⇢ and action a finishes when overall frustration level becomes equal or greater than a certain threshold value. This value is determined based on a tolerance formula:
T olerance = e|| Ov||⇤ ' where, ||Ov|| denotes the translation of the object on the table because of the collision with the end e↵ector and ' the number of trials from the beginning of learning.
Additionally, the online learning process may also end when the following criterium has been met: level of frustration favours more explorative behavior, whereas low or high level of frustration leads to a more exploitative behavior. For the purpose of our simulations, frustration was represented as a simple leaky integrator: df dt =
L ⇤ f + A0 where, f is the current level of frustration, A0 is the outcome of the action (success or f ailure) and L is the fixed rate of the ’Leak’.
In Eq. 5 the ’leak’ rate (L) was fixed and kept at value 1 for all
simulations [
F rustrationLimit = e1/p n where, n refers to the number of grasp pairs. (5) (6) (7) (8) 3.4
The main focus of the presented work is investigating an e↵ect that
impulsiveness has on frustration level and on learning. The learning rate and the speed
of decision making is an important issue in human-robot interaction [
As mentioned before, the candidate grasping points are first determined in simulation, and then transferred to a robotic arm for real-word experimentation. V-REP simulator is used as the simulator and the Cyton-Veta Robotic 7-DOF robot arm by Robai (shown in Fig 3) is used as the experimental platform. The reachability of the arm is about 45 cm. Also in the experiments, we used three objects of di↵erent size and shape (i.e., a small cubic plastic block, a plastic bowling pin and a spherical plastic ball). We compare the performance of four (a) Success lengthwise on the block.
of (b) Success of grasp transverse grasp on the pin.
(c) Failure lengthwise on the ball.
of grasp di↵erent action selection methods discussed in the previous section. A high value of impulsiveness results in a faster increase in a frustration level (in other words, in a “short-tempered” agent). For comparison reasons, we use here two di↵erent values of impulsiveness: a low value of 0.01 and a high value of 100. The results of our experiments support our proposed hypothesis. An agent with low impulsiveness spends more time on exploration, testing more grasp pair possibilities than an agent with a higher value of impulsiveness. For demonstration purposes, we chose three di↵erent objects that vary in their graspability properties: a cube that is relatively easy to grasp, a plastic bowling pin that is easily graspable but it is liable of toppling down, and finally, a ball that is not graspable at all. We compare the decision and learning rate of the robot that uses our proposed strategy (M4) with the one based only on frustration (M3). Fig. 4 shows robot’s level of frustration for each learning epoch while the robot was learning how to grasp the block. The 84 possible grasp pairs generated in simulation were used in a real world scenario. Since the robot can easily grasp the cube, the frustration level is kept low and the learning process terminates before it reaches its limit value, 1.115 (i.e., according to Eq. 8.). In case of the pin (see Fig. 5), the simulation generated 116 possible grasp pair candidates that were subsequently used by the robotic arm. Since the pin is quite light, the arm pulls it down for some grasp pairs. When the pin fells down, the frustration threshold is decreased for the related grasp pairs according to the Eq. 7. Hence, the robot learns that these grasp pairs should be eliminated from the set and immediately proceeds to test another grasp pair. While for some grasp pairs grasping of the pin was possible, the robot was not able to grasp the ball for any of grasp pairs. The ball was made of a hard plastic material and quite light, so every robot’s attempt to grasp it resulted in a ball rolling over on the scene Fig. 3(c). After each trial, the robot’s tolerance for frustration decreased rapidly resulting in that the robot switches to another grasp pair. With each failure, the overall frustration level was raising and quickly exceeded the tolerance threshold (that at the same time was being decreased). Although 92 grasp pairs were transferred to the real world scenario, only after a few steps the robot learned that the object is not graspable. Fig. Fig. 5. Frustration Rate Changes For Pin Grasping with Methods M3 and M4. 7 shows the comparison of the results for all four strategies of action selection. The frustration-based action selection methods require a lower number of trials to learn the graspability of the objects compared to the standard softmax action selection with fixed temperature parameter and "-greedy action selection. The agent with higher value of impulsiveness performs slightly better than the agent with low value. 5
We have presented our intrinsically motivated reinforcement learning system for learning graspability of novel objects. Intrinsic motivation is provided by frustration-based action selection methods during learning, and tolerance values are determined based on impulsiveness of the robot. Our claim is that impulsiveness can be adjusted based on the task that the robot is executing. We have analyzed this mechanism on a robotic arm to learn graspability of di↵erentshaped objects. Our results reveal that the intrinsic motivation helps the robot learn faster. Furthermore, the decision on graspability is made earlier by taking impulsiveness into account. Our future work includes extending the experiment set and investigating impulsiveness parameters in detail for di↵erent domains with varying time constraints.
This research is funded by a grant from the Scientific and Technological Research Council of Turkey (TUBITAK), Grant No. 111E-286. TUBITAK’s support is gratefully acknowledged. We thank Burak Topal for his contribution for robotic arm movement and also thank Mehmet Biberci for his e↵ort on vision algorithms.