Reinforcement learning (RL) is a key method for providing robots with appropriate control algorithms. Controller blending is a technique for combining the control output of several controllers. In this article we use on-line RL to learn an optimal blending of controllers for novel faults. Since one cannot anticipate all possible fault states, which are exponential in the number of possible faults, we instead apply learning on the e ects the faults have on the system. We use a quadcopter pathfollowing simulation in the presence of unknown rotor actuator faults for which the system has not been tuned. We empirically demonstrate the e ectiveness of our novel on-line learning framework on a quadcopter trajectory following task with unknown faults, even after a small number of learning cycles. The authors are not aware of any other use of on-line RL for fault tolerant control under unknown faults.
One of the most important uses of reinforcement learning (RL) is for controlling
robots, using reinforcement learning control (RLC). It has been shown, e.g., [
Model-free RLC assumes that the control system starts with no model and solves the Bellman equation based on running experiments with appropriate rewards, to create a matrix of values that serves as the model. Although modelfree RLC can prove accurate, its main disadvantage is a long convergence time. Since the policy space for robotic interactions can be extremely large, RLC requires a large number of iterations to achieve convergence. In addition, if the plant is unstable (as is that of a quadcopter), or safety is an issue, using RLC can prove di cult in practice.
The alternative approach is to use model-based RLC, which is also known as iterative learning control (ILC). ILC re nes the reference or input signals of a desired maneuver based on data from previous executions. This can be used to update model parameters or extend an existing model.
We assume that we have modelled the quadcopter with a linear
approximation of the underlying non-linear ight dynamics. Further, we assume that
unmodeled dynamics (in our case, faults) can be represented as linear
multiplicative term to the actuation dynamics. Given planned trajectory inputs uk,
each ILC iteration can be decomposed into two steps: (1) disturbance estimation,
where a Kalman lter computes the current estimate of the disturbance ; and
(2) input update,where we compute an improved quadrotor input, uk+1. Using
this framework, the input can be abstracted at any level; e.g., from robot thrust
and angular velocities [
The majority of applications of RL assume that learning is done o -line. However, there are many situations in which a robot will encounter novel situations and needs to adapt to those situations. We address that scenario in this article. In particular, we examine a quadcopter that has been pre-programmed with a set of controls and uses control blending to operate within a known control environment. However, for novel scenarios the robot must adapt. We introduce novel actuator faults into a quadcopter, and use ILC to learn new control laws.
We have de ned the quadcopter to have a hierarchical control architecture. The lower-level controllers use PID methods to control each of the quadcopters three axis of movement. The higher-level controller uses blends of the lower-level controllers to control ight trajectories. We have pre-de ned a set of control laws for nominal and fault modes. We then subject the quadcopter to unseen fault and then allow the quadcopter to repeat the novel conditions for the unseen faults to learn new high-level control laws.
This article proposes the rst use of ILC for learning novel fault-tolerant control laws. We empirically demonstrate that we can learn new controls from a small number of learning trials. 2
This work builds on extensive prior work in RL, fault-tolerant control (FTC) and Fault Detection and Isolation (FDI).
A signi cant body of work exists for RL for robotics applications, dating
from [
In the area of FTC [
Fault-Tolerant Control (FTC) can be divided into two types, passive and
active. Passive fault tolerant controllers are designed o -line against prede ned
models for certain operating conditions and have no ability to react to
unanticipated faults. Passive FTC enables fast adaptation to faults, within the prede ned
operating conditions. Active FTC uses on-line data to recon gure the controller
to stabilise the plant. For a comprehensive study between the two approaches
see [
Reinforcement learning (RL) [
Reinforcement learning learns a policy : S ! U , which de nes which
actions should be taken in each state. Q-learning [
Q(st; ut)
Q(st; ut) + rt+1 + max Q[st+1; ut] u
Q[st; ut] where rt+1 is the reward observed after performing action ut in state st, is the learning rate (0 1), and st+1 is the state that the agent transitions to after performing action ut. After Q(st; ut) converges, the optimal action for the agent in state st is arg maxu Q(st; u):
In Q-learning and related algorithms, an agent maintains a table of Q[S; U ] based on its history of interaction with the environment. An experience hs; u; r; s0i provides one data point for the value of Q(s; u). The data point is that the agent received the future value of r + V (s0), where V (s0) = maxu0 Q(s0; u0); this is the actual current reward plus the discounted estimated future value. This new data point is called a return. The agent can use the temporal di erence equation to update its estimate for Q(s; u): or, equivalently,
Q[s; u]Q[s; u] + (r + max Q[s0; u0] u0 This section presents a summary of our results. We have shown that RL can be used to dynamically learn how to stabilise a robot's trajectory given previously unseen faults. We used a quadcopter to demonstrate how path-following deviations can be reduced following learning epochs.
We will rstly present the nal Q-Matrix (Figure 1) that was learned. Positive and negative Q-values represent good and bad performance respectively. There is a large amount of negative values which can be attributed to our reward function, see Section 5.1. We say the matrix has converged perfectly if only a single positive Q-Value exists per row. This is not evident in this matrix but with adjustments to the reward function and enough learning epochs this matrix would converge. Another interesting point to note is the magnitude of the QValues. The rst row has signi cantly higher values than the rest. This is because the rst row represents the lowest deviation rate and this row will get trained for every fault since for the deviation rate to increase to the higher partitions it must rst go through rst partition. To see the improvements in trajectory following performance for a quadcopter, we compared the RLC with the original quadcopter con guration. As far as the authors are aware there is currently no baseline controller for unknown faults to compare against. In future we intend to run comparisons against other FTC architectures. We use total trajectory deviation in centimetres to gauge how much the tracking accuracy improved. Table 1 shows this comparison for several magnitudes of unknown rotor faults. We denote the number of learning cycles of RLC using . We noticed a 63-75% decrease in the trajectory deviation after a small number of learning cycles.
Figure 2 shows the time it took to re-stabilize the quadcopter along its trajectory with acceptable deviation after each completed learning cycle. It is clearly visible that after a small number of learning iterations the time to re-stabilize drastically reduces but never converges to 0. The spikes in the graph show an increase in the magnitude of a benchmark fault induced.
This section describes the mapping of the RL approach to our experimental
domain, i.e. what the RL tuple hS; U; T; R; i means for our quadcopter domain.
For the experimental set-up we used an opensource Matlab implementation of a
quadcopter ight simulation [
Our objective with this simulation is to implement on-line reinforcement learning for faults of unknown origin and magnitude. As far as the authors are aware not much work has been done using RL to learn a blended controller for unknown faults on-line. The simulations core task is trajectory following which is de ned as a list of time indexed way point locations in the form of a triplet (X , Y, t), presenting desired X and Y position at time t. Given the quadcopters current position at time t as (x, y) the deviation from the trajectory can be computed as: t = q (X x)2 + (Y y)2
The trajectory used is a square and can be seen in Figure 3 Left. We de ne a single learning cycle for RL as a single execution of the shown trajectory with one unknown faults. The faults are injected at the same point in the trajectory for consistency. After every learning cycle we allow for time to fully stabilize along its trajectory so no residual e ects from previous learning cycles will in uence future learning quality. Since we have a cyclic trajectory we can continuously repeat the learning cycles and analyse the performance improvements over time and on-line. Each learning cycle can be de ned in terms of four phases, which are: 1. Stabilize - Ensure nominal operating conditions. 2. Learning - Random fault in speci ed range and stabilizing using RL. 3. Stabilize - Ensure nominal operating conditions. 4. Evaluate - Test against a benchmark error to check improvement.
The Stabilize phases simply ensure the quadcopter returns to its trajectory with acceptable deviation rate and in nominal operating conditions before proceeding to the following phases. The Learning phase consists of random fault injection and stabilizing with RL. The Evaluation evaluates the current learned policy against a benchmark fault to record the improvements learned over time, Figure 2.
RL States The space of unknown faults is simply too large to apply learning to each fault individually. We hence classify the states S for our RL implementation in terms of the e ect that unknown faults have on the trajectory following task. For this metric we chose the deviation rate from the desired trajectory, which we de ne as: = d X4 dt i=0
t i;
M = X 'i i:
i=1 where i is the trajectory error between current and desired position at time step t i. We then de ne S as a partition over the parameter space of into N regions of equal size.
RL Actions Our action space U is de ned in terms of blended control for which we use a linear combination of prede ned controllers. In other words, given a set of M prede ned controllers = f 1; ; M g and corresponding weights f'1; ; 'M g, our applied blended control is given by where PM
i=1 'i = 1. We de ne our action space U as a partition of size P over the parameter space of '. The granularity of this partition will dictate how many di erent blended controllers are being learned on.
We then de ne our Q-Matrix as Q(S; U ) and set N and P to 5 and 10 respectively. In other words we are learning the optimal blended controller for each deviation-rate sub-partition. The size of the matrix increases drastically with the size of the partitions. We hence chose small enough partition sizes to concentrate the learning into a smaller region. The transition function T maps the current state and action chosen to the new state. In our context T is already given by the simulation itself.
For RL to be able to learn on-line we must have some performance metric to evaluate how well a controller performed during the fault recovery for each separate learning cycle. For this metric we choose the time, , until the quadcopter is stabilized on its desired trajectory. (1) RL Rewards Since we use as our performance metric we must also compute a baseline value to compare this against, which indicates positive or negative reward for the blended controller used. Since we have no prior knowledge about the faults, we compute , a running average of . For each subspace of Si we de ne i to allow larger errors more time to stabilize. We assign credit based on the performance against the running average.
C =
Credit Assignment Problem The Credit Assignment Problem [
R(Si) = C t where t is the duration that was in the bounds of Si and C and are as previously de ned.
For this implementation we do not use , the future expected reward. Estimating the expected deviation rate t+1 is challenging as the controller is synthesised on-line and the faults are unknown. In future work we will extend the reward function to include the term.
Error Generation We focus solely on unknown rotor faults for this article for simplicity but this framework works for any unknown faults that cause trajectory deviations. More speci cally, we use a multiplicative term on the rotor speed to represent the unknown fault. To achieve an even distribution of learning we randomize , such 0 , where is the upper limit on the fault. We incrementally increase after a number of learning cycles. This will give a larger variance of deviations allowing the system to apply learning across all sub-partitions. Figure 5 shows the rotor fault magnitudes used over 250 learning cycles. Note this gure shows the benchmark errors used as well as randomized errors for training.
Simulation Details For completeness we will give a full list of low level controllers and other parameters used for the experiments. The PID controller coe cients for nominal and fault (indicated by superscript N and F) controller for each control axis (indicated by subscript ; ; ) can be seen in Table 3. It is worth noting that these controllers are not tuned for any speci c fault; they are simply tuned with a more "aggressive" coe cient. Figure 1 shows the matrix after 250 learning cycles. We set the initial value for to 3% and increased this after every 50 learning iterations up to 7%. We apply the fault to the same rotor every time. In this article we presented a novel reinforcement learning approach for online, real-time learning for unknown faults. We empirically demonstrated the e ectiveness of this approach on a Quadcopter trajectory following task. We noticed a 63-75% decrease in the trajectory deviation due to an unknown fault after a small number of learning cycles. In this set of experiments only rotor faults where investigated but given appropriate controller pairs for blending the authors believe this approach will work for other system disturbances such as wind or sensor faults. Since we are learning on the disturbance space ,instead of the fault parameter space, this learning approach will work for most faults that cause a path deviation. The novelty of this approach is that the learning phase can be conducted on-line and needs very few iterations before converging compared to traditional RL methods. Further more, we are not aware of any other RL based approach for FTC under novel faults. This could provide adaptive FTC capabilities to systems that are not easily xable or recon gurable such as satellites or space rovers. Our future work includes focusing on improving the learning process and attempting training on a multitude of errors. For simplicity, this article also only explores a two controller blending strategy but in theory this is not a limitation. Larger sets of prede ned controllers for blending can be trained using our described method but with a considerably larger number of training phases which will also be explored in future work.