Autonomous drones typically have limited battery capacity, which presents a major challenge for persistent, long-term deployments. In the context of water quality monitoring, aquatic drones must navigate rivers and lakes to collect real-time data, where the battery consumption is heavily influenced by dynamic environmental factors such as flowing current and wind. Intelligent battery management is a requirement for the success of these missions. We propose a formalization of the battery management problem in terms of Partially Observable Markov Decision Processes (POMDPs). We model the problem as a POMDP where the agent modulates the energy provided to its propellers. The drone can estimate the “difficulty” to traverse an interval by observing the environment. An important aspect of this formalization is the prediction of future intervals' difficulty values based on current observations by exploiting a priori geometric information, which we call Task Difficulty Propagation (TDP). We investigate variations of this approach and analyze related performance.
Autonomous aquatic drones are increasingly used for water monitoring. A key factor
for the success of data acquisition campaigns is mission awareness, which is composed
of three main elements [
Existing approaches for estimating the battery consumption in autonomous systems
can be divided into two main categories, namely decision [
We apply our method to the following case study: given i) a path made of four
perpendicular segments (Figure 1.c), in which each segment has a specific degree of
difficulty (i.e., a measure of the forward speed achieved for a given amount of energy
consumption) and ii) an initial amount of battery energy, the agent has to make decisions
about the amount of engine power needed at each time instant to minimize the time
spent to reach the end of the path and maximize the probability to complete the mission
before battery depletion. To this end POMDPs are extended with a novel simulation
strategy called Task Difficulty Propagation (TDP) which is based on Markov Random
Fields (MRF), and allows to represent the dependencies between segment difficulties
and to propagate this information for improving the belief over the difficulties in the
entire path. First experiments show promising enhancements to mission performance.
Future developments aim at using the method on the INTCATCH [
The goal of an agent based on POMDPs is to maximize its expected total reward E[Pt1=0 tR(st; at)], by choosing the best action at in each state st at time t.
Figure 1.d shows the proposed POMDP for battery management as a graph, where states are represented by circles, actions by rectangles and conditional dependencies between states by arrows. The states are: i) battery voltage interval B 2 fb0; b1; b2; b3g, where b0 represents empty battery, ii) speed interval V 2 fv1; v2; v3g, iii) distance to path end D 2 fd0; d1; :::; d8g, where d8 is the starting interval of the path (i.e., the maximal distance) and d0 is the end point of the path (i.e., the goal), iv) time interval T 2 ft1; t2; t3; t4; t5g, where t1 is the time interval in which the path starts, v) current segment S 2 fs1; s2; s3; s4g, vi) difficulty of segments F1; F2; F3; F4 j Fj 2 fL; M; Hg, where L=low, M=medium, H=high, and difficulties may depend on water flow or other environmental factors, vii) difficulty of current segment Fc 2 fL; M; Hg, viii) engine power P 2 fL; M; Hg, which corresponds to a uniform discretization of the combined effort of the drone’s propellers. ( 2413 ; : : : ; 2413 ).
States B; V; D; T and S are observable, while states F1; : : : ; F4 and Fc are hidden,
hence the model has a mixed observability [
In order to represent dependencies between segment difficulties we define two elements: i) a cosine similarity matrix which provides a measure of geometric similarity between pairs of segments in the path (see matrix K in Figure 1.c), ii) a pairwise MRF to compute a joint probability function over configurations of segment difficulties (see the graph in Figure 1.c). The MRF is used to factorize the joint probability of segment difficulties as a product of potential functions over the maximal cliques of the graph. The joint probability is then written as p(f j ) = Z(1 ) Qq2Q q(fqj q), where f is a vector of difficulties (e.g., (H; M; L; L)), is a parametrization of the MRF, Q is the set of (maximal) cliques of the graph, q(fqj q) is a potential function for clique q and Z( ) is a normalization factor called partition function. We use a simplified version of MRF called pairwise MRF in which the parametrization is restricted to the edges of the graph rather than to the maximal cliques and we conveniently express potentials as exponentials with Boltzmann distribution. In this way the product of potentials in p(f j ) can be computed by adding the energies of each of the maximal cliques. Since our difficulty variables are discrete, we represent the energy functions as tables of (non-negative) numbers representing the relative “compatibility” between the different assignments of segment difficulties. In particular, given a pair of segments (si; sj ) j i; j 2 f1; 2; 3; 4g having cosine similarity equal to 1, 0 or -1, the energy functions of the corresponding pair of difficulties (fi; fj ) j fi; fj 2 fL; M; Hg are defined as E1(v; w), E0(v; w) and E 1(v; w) where v = I(fi), w = I(fj ) and I( ) is the index function. To compute the energy function of couples of segments having generic cosine similarity k 2 [ 1; 1] we interpolate the parameters in E1(v; w), E0(v; w) and E 1(v; w) by quadratic polynomials.
The joint probability computed by the MRF is finally used to update the beliefs generated step-by-step by the TDP simulator according to the formula btj = btj + (1 ) btj p(f j ), where t 2 N is the time step, j 2 f1; : : : ; 243g is the index of the difficulty configuration (i.e., hidden state), 2 [0; 1] is a weighting factor, btj is the probability of the j-th difficulty configuration provided by standard simulator, p(f j )) is the joint probability of the j-th difficulty configuration provided by the MRF, and btj is the probability of the j-th difficulty configuration provided by TDP simulator. 3
We implemented the POMDP model in a POMDPX XML file [
Subsequently, we modified the pomdpsim simulator in two ways: i) by forcing the belief to the true hidden state (i.e., the probability distribution has all the mass into the real state and zero probability for other states), we call this simulator oracle (ORC) since it has complete information about the true hidden state, ii) by implementing the TDP approach presented above, we call this simulator TDP since it considers path geometry to improve the belief update over segment (i.e., task) difficulties. We performed 100 simulations by ORC and TDP simulators, respectively, and compared results with those achieved by STD simulator, in terms of belief evolution, average number of steps to reach the goal, and average number of battery failures.
Figure 2.a shows the evolution of beliefs in the three simulation cases: columns
represent different simulation strategies, namely, STD, TDP and ORC, while rows
represent segments 1, 2, 3 and 4 of the square path. Each chart shows the average
belief (over 100 simulations) for a particular simulation strategy in a specific segment.
A consequence of the improved belief update strategy in TDP is the improvement of
performance in terms of (reduced) number of steps to reach the end of the path. This is
observable in Figure 2.b which shows the average number of steps required by each
approach to reach the end of segments s1; s2; s3 and s4. We observe that using TDP (blue
line) the agent is able to reach the end of the path with less steps than using STD (red
line) and with a similar failure rate (i.e., the percentage of times in which the battery
ends before the end of the path). ORC has the best performance with 96.4 steps, on
average, to reach the end of the square and 24% of failures. The second best performance
is that of TDP with 102.1 steps and a failure rate of 22%. Both approaches outperform
STD which needs 111.6 steps and has a failure rate of 22%. A key extension of the
proposed approach concerns scaling on more complex scenarios. To this end we are
currently exploring the usage of Partially Observable Monte Carlo Planning (POMCP)
methods [
This work is partially funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 689341.