Introduction

Learning is a process to improve the performance of a system based on its past experiences. This method occurs when the problem seems too complicated to solve in real time, or when it seems impossible to solve the problem in a classical and rigorous. An example of learning methods cited reinforcement learning.

The reinforcement learning is a technique which is to acquire the agent executor behavior desired by methods based on the concept of reward or punishment. The optimal behavior of an agent is often difficult to implement given the large number of variables that may play a role. In the framework of reinforcement learning, the agent can learn to behave in the same way as we learn to ride a bicycle from a signal known as reinforcement. One of the fundamental parts of the system of reinforcement learning is the Q-function; it allows the agent to learn how to choose good actions and how to measure their utility.

One of the goals for the navigation of autonomous mobile robots is the avoidance of obstacles, several techniques and methods are used for this reason [1], [2]. The algorithm, which allows the mobile robot starting from a stop position and a final position following a pattern set meadows. If the robot encounters obstacles in its path, the algorithm of obstacle avoidance takes control of mobile robot. Once the path of the robot is free shipping to the destination is taken [3]. In this article we solve the problem of navigation with obstacle avoidance by a method of learning. For this purpose the Q-Learning with the Q-function is generated by a network of neurons 2 Reinforcement learning

Principle

The reinforcement learning is a learning technique based on trial and error, and the interaction between the agent and its environment [4], [5], where from a state or situation s in the environment, the agent selects and performs an action that causes a transition to state s'. He receives in return a reinforcement signal r, which can be a reward or punishment. The purpose of this learning is to maximize future rewards [4]. Fig. 1 shows a state of interaction between the agent and the environment

Q-Learning

The Q-Learning is a famous algorithm that is used for solving problems of reinforcement learning, it was proposed in 1989 by C. J. Watkins [6], [7]. This algorithm is based on three main functions, an evaluation function, a function of strengthening and reinforcing function.

Evaluation function

The state of the environment and the action performed by the agent are evaluated by this function is called Q-Function From a current state s of the environment that is observed by the agent, an action is performed based on knowledge available within the internal memory (This knowledge is stored in the form of utility value associated with a pair of "State Action"). [8]

Reinforcement Function

After executing the action by the agent in its environment; Reinforcement function of R, provides for each new state s', r a signal that can be a reward or punishment, this signal usually takes a single value, 1, -1 or 0, which is used by the Update function to adjust the Q-value function associated with the pair "State Action" [8].

Update Function

This function uses the value of reinforcement to adjust the value associated with the pair "State, Action" which has just been completed [8].

The Q-Learning as a principle estimating the Q-function noted by:  Q , and is defined by:

          ' , ' max , * ' 1 * 1 * a s Q r E s V r E a s Q a t t         (1)

Using the equation for updating one finds that:

            ' , ' max , 1 , ' 1 1 a s Q r a s Q a s Q t a t t t t t t t t t             (2)

Or:

1  t

r Is the reinforcement received when the agent selected the "a" action in the state "s" to move to the state "s'".

t  Is a real positive:   1 , 0  t 

In principle it should be randomly explore the environment for a large number of iterations for the Q-Learning converges to the optimal Q -function [8], then we can use the optimal policy defined by:

    a s Q s A a , max arg * '    (3)

System architecture of reinforcement learning

The objective of this proposition is to have a learning system that allows a robot that moved in an environment that is totally unknown in avoiding obstacles.

The generation of the Q-function is performed by an artificial neural network-type MLP, where the inputs are reading sensors associated with the robot on its three sides, giving the perception of its environment, and the vector of possible actions (Turn Left, Forward and Turn Right). The choice of action that the agent must perform is a function called by function selection.

Generating the Q-function with an Artificial Neural Network

The generation of the Q-function can be made by a table in which each cell corresponds to an approximation of the Q-function for a configuration of the pair state / action. This severely limits the size of problems we can solve; in fact, many realworld problems such as robotics have a large space. An innovative approach to the generation of the Q-function in the case of large spaces is to use Artificial Neural Networks. The approximation of the Q-function is obtained, using Artificial Neural Networks with the learning algorithm Back propagation [9], [10] In this implementation, the Artificial Neural Networks chooses is a Multi Layer Perceptron, which entered as the state of the environment and the possible actions, a layer containing n h hidden neurons and one output neuron that max of the Q-function [11]. The activation function of all neurons is the sigmoid function.

                    et t t t t t t Q S a w Q S f w a s Q S a a s r t w t w arg 2 2 2 , , , 1           (4)

 For the vector of means we have

                    et t t t t t t Q S a w Q S f w a s Q S a a s r t b t b arg 2 2 2 , , , 1          (5)

Where: Q target is simplifying the equation of optimality BELLMAN which is given by the following equation

    w a S Q a s r Q t t t                      S w w f S w a s Q S a a s r t w t w t t t t t 2 1 1 1 , , , 1   (7)

 For the vector of means we have:

                      S w w f S w a s Q S a a s r t b t b t t t t t 2 1 1 1 , , , 1   (8)

With:

f : Activation functions of neurons in the hidden layer. S : State of the environment.

 w a s Q t t

, , : Function value state / action corresponding to the action performed.

These changes in values for the weight matrix and vector of bias are present if the reinforcement signal is: -1 or 0.

Function Selection Action

The neural network allows us to generate the Q-function. The set of possible actions is given by where:

• a 1 : Turn Left Action.

• a 2 : Forward Action.

• a 3 : Turn Right Action.

The selection of the action that the robot must execute is based on the policy Exploration / Exploitation (EEP) [12], for this reason we used the greedy method ( greedy) which consists of choosing the greedy action with probability  and to choose a random action with a probability,

  1 . •   1 , 0  p a random number. • If   p

we chooses a random action a "Exploration", where A a  of the all actions possible.

• If

  p you selected     w b S Q Arg S a A b , , max   "Exploitation"(9)

Environment

Reading the state of the environment is done through sensors that are placed on three sides of the robot. Two on the left, two on the right and three in front. The Simulation of the navigation of a mobile robot by the Q-Learning using artificial neuron networks 7

sensors can be used type of proximity detection. The opening angle of each sensor varies between -π/12 and π/12. The state vector S is chosen so as to obtain information on the existence of obstacles on three sides of the robot. This vector is composed of seven binary variables s i , i = 1,.. 7. The choice of these variables is made in order to have any information or anything. i.e., for example if s i = 1 then there is an obstacle near the robot, if s i = 0 no obstacle near the robot.

Fig. 5 shows a state of the environment that gives the value of state vector S, such as

  0 , 0 , 0 , 0 , 0 , 1 , 1  S .

Model of the Robot

The type of robot that we have chosen for the application is circular of radius R. This robot is operated by two independent wheels separated by a distance L. Fig. 6 shows this type of robot.

                      2 cos 1 k k k v k x k x r r   (11)                       2 sin 1 k k k v k y k y r r  (12)

x r (k) and y r (k) are the x and y of the robot in the landmark (Ox, Oy).

The orientation of the robot      k k k       1(        k k k v l r    2        k k L k l r       2 1 (15)

Where: L: is the distance between the right wheel and left wheel.

Algorithm Q-Learning with Artificial Neural Networks

(1)-Initialize random weights W of the neural network;

(2) Give the initial position of the robot [(X r (0), Y r (0), θ r (0)];

For k = 1 to iteration k

(3)-Reading the S t state of the environment by the seven sensors (L, A, R);

(4) For a fixed state, calculation of the Q-function by the neural network for the three possible actions (TL, Ar, TR);

(5)-Determination of the optimal action is action which is the maximum value of Q (optimal_Action  Max (Q));

(6)-Run the optimal with probability  ;or random action with probability 1- (7) Reading the new state S t+1 of the environment through sensors the Seven (G, A, D); To test the ability of our artificial neural network proposed in Fig. 10 shows a change of environment, where the robot moves through its environment while avoiding obstacles.

Conclusion

In this paper we presented the technique of reinforcement learning where we have chosen the Q-Learning algorithm by using artificial neural networks for the