-We report on several test experiments with a mobile agent equipped with an artificial neural net control to achieve a basic route direction goal reflex in a 2-dimensional environment with obstacles. A real assembled 4tronix Initio robot kit agent is reproduced with its sensor and motor characteristics in a virtual environment for experimenting and comparing the behavior of its artificial neural net control with two different learning approaches: A standard supervised error back propagation training with examples, and an unsupervised reinforcement learning with environmental feedback.
Obstacle avoidance is a basic task for mobile agents.
Commercial and research applications address the problem by
using state of the art adaptive techniques and mathematical
modeling. In this work we confronted with the task of using
a simple neural net architecture to equip a real 4tronix Initio
[
As a result we identified a simple artificial neural net architecture and a pair of training strategies that would let the agent show simple adaptive/reactive capabilities in avoiding obstacles, while achieving a given target position goal. The agent’s control neural net model is deliberately kept at a reflex level state, the perceptron basic input/output transduction, with no high level onthologies or primitives for describing the environment, no environment model or map acquisition capabilities, and no planning ability of any sort. The environment only exists for the pattern that it induces on the agent sensors at a given position and orientation. The agent will have to react by issuing control settings to its motor components, taking into account its given goal.
The agent hardware is a 4tronix Initio [
The virtual environment is constructed as a bit matrix of dimension m × n. Each bit represents a virtual position for the agent. A bit is set to 1 if the position is blocked, there is an obstacle, and the position cannot be occupied by the agent. The goal area of the environment is a circle identified by a center position and a boundary radius. The virtual agent emulates the real hardware in performing its basic control commands to the hardware controllers. The basic functioning cycle has three steps: Sample sensors, Compute orientation and velocity, Run for a time unit.
The artificial neural net (ANN) is emulated through the
use of the Neuroph [
The experiments are described when the ANN is trained according to two different protocols: Supervised error Back Propagation (BP), environment reinforcement learning (RL).
1) The Supervised error back propagation protocol: The BP protocol was experimented with training sets (TS) constructed in two different ways, the first (BPp) by sampling the control choices of a human pilot, the second (BPh) by recording the behavior choices of a heuristic evaluation function in a wide range of enumeration of input sensor patterns. The use of BPh allowed the easy construction of much larger virtual TS, while TS construction with BPp required synchronized reading the agent sensors and sampling the human pilot, who on the other hand was allowed to comprise higher level cognitive faculties, and was allowed to look at the environment map while making his decision.
To preserve generalization capabilities and avoid overfitting of the ANN, the training process was stopped at predetermined network mean square error limit values. For this purpose, each training sets were split in two subsets of training subsets and test subsets, containing respectively 90% and 10% of the original TS samples. Optimal limit network error values were determined by training the network on training subsets and testing network response on the test subsets: when the error on the test subsets became stationary or increasing, training was paused and finally completed by using the full TS and the minimum network error limit.
After the training process, the trained ANN’s were tested on the agent control system and some critical aspects emerged as of the dimensional insufficiency of the TS obtained with BPp when compared to the dimension of the inputs state configurations. It did, in fact, bring about insufficient network output response polarization on new input patterns and a random-like behavior of the agent in specific configurations. On the other hand the ANN’s trained with TS constructed with BPh was often trapped in stationary or cyclic behavior in sub-optimal positions with respect to the navigation goal. In consideration of the complementary critical aspects described for the BPp and BPh cases, a third instance of the ANN was trained with an incremental training process that combined both pilot driving and heuristic evaluation TS. In this case the learning process consisted in two training phases. In the first phase, the BPh TS was used for training the ANN for a small number of training epochs in order to give the network base response capabilities in covering a wide range of input configurations. In the second phase, the BPp TS was used until training completion. As the incremental trained network was finally tested on the robot, the critical behaviors were relieved.
All the test results reported in the following are obtained by running the net configuration obtained at the end of the training process as described, with a sample environment problem.
2) The Reinforcement Learning protocol: The same net
architecture has been used with an unsupervised reinforcement
training protocol Q-learning [
The experimentation was performed after training the ANN both with the BP protocol and with the Q-learning protocol leading to two net configurations named SUPERVISED (SU) and REINFORCED(RE) respectively.
The RE network was trained for 4000 learning sessions of 100 base cycles. For each session a random start and target is assigned in a randomly generated environment. Main learning parameters are set by an empirical optimization process to the following values: Learning rate= 0.036, Future actions discount factor g= 0.24, and Stochastic action selector temperature T=4. A high temperature T value is necessary during the learning process to maximize reinforcements. The T value is subsequently lowered on tests to 0.4 value to appreciate neural network response and control system behavior.
On the other hand, SU is trained for one learning epoch on the BPh TS, containing 1.152.000 training samples, and resulting in a 0.14 mean square error after training. The following 2076 training epochs are performed with the BPp TS, with 400 training samples. Mean square error at the end of the training process is 0.07. Both BPh and BPp training sets are generated on several base generation sessions of 50 iterations each. Network weights are initalized with random values in the [-0.02, 0.02] interval.
After training the nets are tested on a battery of several tests on the same environment, organized in groups of increasing environment complexity.
Their behavior, while trying to achieve the target area goal, is sampled as reported in fig. 2 and in fig.3. The whole experiment ranges over 8 batteries of 10 random problems in the same environment, for 10 different random choices of start/target positions. In the figures we show the outcome of just two test batteries, where each row collects an order preserving under-sampling of vfie out of the ten snapshots in the battery, each representing the trajectory of the agent behavior in the same environment. Snapshots of RE and SU behavior in test problems are in the bottom two rows. The top two rows records the agent behavior when controlled by an UNTRAINED (UN), randomly selected net configuration, and when controlled by a Q-learning ADAPTIVE (AD) net. AD is always in learning phase, it is randomly initialized at the first test in the battery, and retains its net configuration through subsequent tests in the battery. As AD gets trained while testing, it is expected to converge to the one of RE.
Each snapshots records the trajectory of the corresponding row agent. Trajectory position points have a time scale color starting from yellow and going to darker red as time passes. Test problems in a row are presented in the same order as they were performed. The order is irrelevant for UN, RE, and SU, but AD’s behavior changes (and improves) while solving a problem. For subsequent tests in the battery, AD’s behavior change gives an idea of how a Q-learning net evolves to a finally trained RE from an UN.
Figure 5 reports a statistics over the tests performed in a single test time of 1600 iterations. The first index indicates ineffectiveness on task achievement, obtained by measuring the time taken for the agent to reach the goal area. High values of ineffectiveness are generally associated to wandering behavior or stationary dead ends encountered during navigation. The second index computes effectiveness and persistence, by measuring time percentage spent inside the goal area after the area is reached. Low values of the index are generally associated with excessive random behavior or fortuitous goal area achievements. The tests are performed on increasing environment complexity levels, and then growing time needed to success. UN network shows negative performances on all tests conditions, while SU network shows the best performances especially into low complexity environments, with a low number of obstacles, moving straight to the goal area in a few number of cycles, basically focusing on target. RE shows best performances particularly in high complexity environments, proving better explorations capabilities and abilities to overcome stationary configurations.
Figure 4 reports performance statistics indexes over increasing learning time of AD neural network from 0 to 400.000 learning cycles, increasing by a 50.000 interval. The progressive trend is evident and demonstrates the adaptive capabilities of the reinforcement learning protocol. Positive performance indexes show a clear increasing trend, while negative performance indexes show a decreasing trend. Performance charts show acceleration between 100.000 and 200.000 learning cycles, with inflection points in this interval, and a final stabilization after 250.000 learning iterations.
We presented simulations of the behavior of a mobile agent equipped with a neural net reflex-like control in avoiding obstacles and achieving a given target position goal. At this stage of the project we use no high level onthologies or primitives for describing the environment, no environment model or map acquisition capabilities, and no planning abilities. We implemented the articfiial neural net control with two different learning approaches: a standard supervised error back propagation training with examples, and an unsupervised reinforcement learning with environmental feedback. We constructed both a real robot agent and a virtual agent-environment simulation system, in order to perform fast and reliable test experiments. The virtual environment let us perform advanced integrated training and test sessions with progressive complexity levels and random configurations, leading to a high grade of generalization for the neural net control. We collected statistical data on several test experiments and compared the performance of the two learning approaches. The analysis of the control system critical aspects and capabilities, as observed in the simulations, favored fixing and improving data presentation in the training protocol. :