This paper presents a type of machine learning is reinforcement learning, this approach is often used in the field of robotics. It aims to determine a control law for a mobile robot in an unknown environment. This kind of technique applies when one assumes that the only information on the quality of actions performed by the mobile robot is a scalar signal which has a reward or punishment, the process of learning is to improve the choice of actions to maximize rewards. One of the most used algorithms for solving this problem is learning the Q-learning algorithm which is based on the Qfunction, and to ensure the generation of this latter function and the proper functioning of the apprenticeship system using an artificial neural network as the statements of changing environments where mobile robots have wide open spaces, the action performed by the mobile robot in its environment is ensured by using a selection function, this action is evaluated by a scalar signal which is -1, 0 and 1.
Learning is a process to improve the performance of a system based on its past experiences. This method occurs when the problem seems too complicated to solve in real time, or when it seems impossible to solve the problem in a classical and rigorous. An example of learning methods cited reinforcement learning.
The reinforcement learning is a technique which is to acquire the agent executor behavior desired by methods based on the concept of reward or punishment. The optimal behavior of an agent is often difficult to implement given the large number of variables that may play a role. In the framework of reinforcement learning, the agent can learn to behave in the same way as we learn to ride a bicycle from a signal known as reinforcement. One of the fundamental parts of the system of reinforcement learning is the Q-function; it allows the agent to learn how to choose good actions and how to measure their utility. One of the goals for the navigation of autonomous mobile robots is the avoidance of obstacles, several techniques and methods are used for this reason [1], [2]. The algorithm, which allows the mobile robot starting from a stop position and a final position following a pattern set meadows. If the robot encounters obstacles in its path, the algorithm of obstacle avoidance takes control of mobile robot. Once the path of the robot is free shipping to the destination is taken [3]. In this article we solve the problem of navigation with obstacle avoidance by a method of learning. For this purpose the Q-Learning with the Q-function is generated by a network of neurons
The reinforcement learning is a learning technique based on trial and error, and the interaction between the agent and its environment [4], [5], where from a state or situation s in the environment, the agent selects and performs an action that causes a transition to state s'. He receives in return a reinforcement signal r, which can be a reward or punishment. The purpose of this learning is to maximize future rewards [4]. Fig. 1 shows a state of interaction between the agent and the environment A c t i o n
S t a t S
R e i n f o r c e m e n t r
The Q-Learning is a famous algorithm that is used for solving problems of reinforcement learning, it was proposed in 1989 by C. J. Watkins [6], [7]. This algorithm is based on three main functions, an evaluation function, a function of strengthening and reinforcing function. Fig. 2 shows the general model for the algorithms of reinforcement learning, by Q-Learning.
After executing the action by the agent in its environment; Reinforcement function of R, provides for each new state s', r a signal that can be a reward or punishment, this signal usually takes a single value, 1, -1 or 0, which is used by the Update function to adjust the Q-value function associated with the pair "State Action" [8].
This function uses the value of reinforcement to adjust the value associated with the pair "State, Action" which has just been completed [8].
The Q-Learning as a principle estimating the Q-function noted by: Q , and is defined by:
Or:
Q* s, a Ert1 V * s Ert1 maxQ* s', a' a'
Qt1st ,at 1t Qt st ,at t rt1 maxQt s',a'
a'
(
In principle it should be randomly explore the environment for a large number of
iterations for the Q-Learning converges to the optimal Q - function [8], then we can
use the optimal policy defined by:
' s arg max Q * s, a
aA
(
The objective of this proposition is to have a learning system that allows a robot that moved in an environment that is totally unknown in avoiding obstacles.
The generation of the Q-function is performed by an artificial neural network-type MLP, where the inputs are reading sensors associated with the robot on its three sides, giving the perception of its environment, and the vector of possible actions (Turn Left, Forward and Turn Right). The choice of action that the agent must perform is a function called by function selection.
Fig. 3 shows the structure of reinforcement learning based on an Artificial Neural
Reinforcement Possible Actions Inputs
ANN
Q Value
Action S. Action 4 Generating the Q-function with an Artificial Neural Network
The generation of the Q-function can be made by a table in which each cell corresponds to an approximation of the Q-function for a configuration of the pair state / action. This severely limits the size of problems we can solve; in fact, many realworld problems such as robotics have a large space. An innovative approach to the generation of the Q-function in the case of large spaces is to use Artificial Neural Networks. The approximation of the Q-function is obtained, using Artificial Neural
In this implementation, the Artificial Neural Networks chooses is a Multi Layer Perceptron, which entered as the state of the environment and the possible actions, a layer containing nh hidden neurons and one output neuron that max of the Q-function [11]. The activation function of all neurons is the sigmoid function. Fig. 4 shows the Simulation of the navigation of a mobile robot by the Q-Learning using artificial neuron networks 5 generation of the Q-function with Artificial Neural Network-type MLP for Q
ed roP t e x itcon iitym P o s s il b e A c it o n s State Vector S L 1 e tf 2 F 3 o rw 4 a r d iR 6 g th 7 5 R A L c
Hidden Layer Output Layer QS, a, w
The learning Network is based on updating or adjusting the weight matrices of the Neural Network using the equation of the update of the classic algorithm of QLearning and the algorithm Back propagation 4.1.1 The output layer For the weight matrix For the vector of means we have w2 t w2 t 1 rst , at aS Qst , at , wt f S Q aS Qt arget w2 b2 t b2 t 1 rst , at aS Qst , at , wt f S Q aS Qtarget w2
Qtarget is simplifying the equation of optimality BELLMAN which is given by the following equation
Qt arg et rst , at max QS , at , w f : Activation functions of neurons in the output layer.
Qst , at , w: Function value state / action corresponding to the action performed.
S : State of the environment.
(
These changes in values for the weight matrix and vector of bias are present if the reinforcement signal is: -1 or 0.
The neural network allows us to generate the Q-function. The set of possible actions is given by where: • a1: Turn Left Action. • a2: Forward Action. • a3: Turn Right Action.
The selection of the action that the robot must execute is based on the policy Exploration / Exploitation (EEP) [12], for this reason we used the greedy method ( greedy) which consists of choosing the greedy action with probability and to • p 0,1 a random number. choose a random action with a probability,1 .
• If p we chooses a random action a "Exploration", where a A of the all actions possible.
• If p you selected
aS Arg max QS , b, w “Exploitation”
bA
f
w1
f
w1
w2S
w2 S
(
For the weight matrix
w1t w1t 1 rst , at aS Qst , at , wt S For the vector of means we have:
b1 t b1 t 1 rst , at aS Qst , at , wt S
f : Activation functions of neurons in the hidden layer.
S : State of the environment.
Reading the state of the environment is done through sensors that are placed on three sides of the robot. Two on the left, two on the right and three in front. The Simulation of the navigation of a mobile robot by the Q-Learning using artificial neuron networks 7 sensors can be used type of proximity detection. The opening angle of each sensor varies between -π/12 and π/12. The state vector S is chosen so as to obtain information on the existence of obstacles on three sides of the robot.
This vector is composed of seven binary variables si, i = 1,.. 7. The choice of these variables is made in order to have any information or anything. i.e., for example if si = 1 then there is an obstacle near the robot, if si = 0 no obstacle near the robot.
Fig. 5 shows a state of the environment that gives the value of state vector S, such as S 1,1,0,0,0,0,0.
For each state in which the robot is found, an evaluation of the action done is found by a signal known as signal reinforcement. This function of reinforcement allows the robot to explore its environment; this signal is related to the values of sensor measurements that indicate the presence or absence of obstacles in three directions, left, front and right, which represent the state of the environment. The value of this function or the signal reinforcement is given by: 1 r 0 1 le robot percute un obstacle
les autres cas. .
le robot avance tout droit
(
The type of robot that we have chosen for the application is circular of radius R. This robot is operated by two independent wheels separated by a distance L. Fig. 6 shows this type of robot. y
Right wheel θ
C Left wheel
L
xr k 1 xr k vk cos k 2k yr k 1 yr k vk sin k 2k xr (k) and yr (k) are the x and y of the robot in the landmark (Ox, Oy).
k 1 k k
k : Is the angular position of the robot in the landmark (Ox, Oy)
vk 1 r k l k
2
With:
r k : The angular velocity of the right wheel of the robot.
l k : The angular velocity of the left wheel of the robot.
(
k 1 2 L r k l k
(15)
(
If r =- 1 (there is an obstacle)
(
(
End if
(
End For 10 Simulation results
The proposed algorithm is implemented for a simulation for a mobile robot in a scene or two obstacles are deferred. The use of the algorithm Q-Learning with Artificial Neural Networks for MLP-type obstacle avoidance is the major objective of our simulation.
The generation of the Q-function is based on the use of Artificial Neural
Networktype MLP, in which learning takes place by the algorithm of back-propagation. The
Artificial Neural Networks is characterized by ten neurons as input cells where seven
neurons present state of the environment and the three other neurons present possible
actions. The values of its input neurons are binary types (
The trajectory followed by the robot during the learning stage is presented in Fig. 8 for 2500 iterations. After learning the trajectory of the robot is presented in Fig. 9 for 2500 iterations.
Simulation of the navigation of a mobile robot by the Q-Learning using artificial neuron networks 11
In this paper we presented the technique of reinforcement learning where we have chosen the Q-Learning algorithm by using artificial neural networks for the generation of Q-function this approach has been used for navigation of a mobile robot in an unknown environment while avoiding obstacles, the results are very satisfactory, and meet the target. This allows us to say that this approach can are used for the navigation of mobile robot in an unknown environment.