This work describes the progress of a research line started two years ago that aims at creating a framework to assess the performance of planning-based autonomy software for robotics. In particular it focuses on an open problem in the literature: the de nition of a methodology for fairly comparing di erent approaches to deliberation, while synthesizing a tool to automate large test campaigns for di erent autonomy architectures under the same robotic platform. We have produced a framework, called OGATE, that supports the integration, testing and operationalization of autonomous robotic controllers. It allows to run series of plan execution experiments while collecting and analyzing relevant parameters of the system under a uni ed and controlled environment. The software platform supports also for the de nition of di erent metrics for evaluating di erent aspects of a plan-based controller. This paper, rst presents the framework capabilities and the methodology to support experiments, then, brie y describes an autonomous controller that follows a timeline-based deliberation, and nally presents some results obtained exploiting OGATE to perform tests to analyze the performance of the controller over a targeted robot.
Modern robotics platforms are becoming increasingly sophisticated and
capable. The deployment of Arti cial Intelligence (AI) planning technologies for
robotic autonomy is considered an important technological advancement to
endow robots with enhanced abilities once addressing real world scenarios. In this
regard, the interleaving of automated planning and execution is a crucial
reference problem for Planning and Robotics research communities. Focusing on
the literature in autonomous controllers, we can observe several approaches
employing di erent technologies for planning and execution { see [
One limitation in current research that is shared by di erent research
initiatives is the rather speci c validation methodologies, experimental settings and
assessment analysis usually performed in a manner that is hardly exportable and
scarcely reproducible on di erent platforms. This lack of methodology leads to
perceive the di erent evaluations more like a \proof of concept" [
The authors current research initiative is dedicated to the design and
development of a software framework to support and facilitate the deployment
of control architectures for robotics platforms [
Paper structure. The rest of the paper is structured as follows. First, we present a set of general metrics applicable to plan-based autonomous controllers and our proposed methodology to deal with their evaluation. In the next section we provide a general view of the OGATE tool, that is able to perform automatic campaigns to evaluate autonomous controllers. A planetary exploration case study and the robotic platform employed to assess experimental campaigns are presented. Then, considering an autonomous controller in the speci c case study, we present and discuss the evaluation of the performance of such controller as a function of the considered metrics within OGATE. Finally, some conclusions end the paper.
One of the contribution of the paper is to de ne a general evaluation methodology for supporting the assessment process of autonomous controllers when applied to a robotics platform. In this regard, a sequence of evaluation steps has been identi ed and is discussed in the following.
In general terms, given an autonomous controller to be assessed, a set of evaluation objectives should be isolated and some speci c performance metrics should be identi ed and de ned accordingly. Then, a set of suitable tests should be de ned and performed so as to collect relevant information constituting a quantitative basis for the evaluation process. Finally, a synthetic view of measurements should be generated, e.g., through PDF reports, to point out the performance of the controller according to the evaluation objectives and metrics de ned in the rst phase. More in detail, the methodology proposed to analyze and evaluate autonomous controllers can be thought as the composition of three sequential phases: evaluation design, tests execution and, report and assessment. Evaluation Design. First, it is required to identify which is the evaluation objective. In fact, according to the evaluation target di erent aspects may result relevant (or not). For instance, measuring the deliberation time or considering the number of dispatched goals in di erent scenarios could provide relevant information about the behavior of the autonomous controller. In this case, very speci c parameters can be considered and analyzed. More in general, a set of parameters applicable to any deliberative system should be considered in order to enable also the possibility to compare performance of di erent control systems in the same operative scenario.
According to evaluation objectives, a metrics de nition task is to de ne parameters that should be measured during execution. This is key as the result of the evaluation strongly depends on the selected metrics. It is important to de ne (at least) a small set of metrics that can be applicable to di erent autonomous controllers. Later in the paper, we will provide a set of general applicable metrics which we exploit in our experiments to assess performance evaluation.
Then, the de nition of di erent scenarios and con gurations to be tested should be implemented. The scenarios can be de ned as the set of constraints and goals that the autonomous controller takes as input. However, to also deal with uncertainty, scenarios should be de ned considering external agents that can dynamically generate additional goals or possible failures that may occur during execution. Such scenarios de nition requires advanced capabilities such as replanning and failure recovering schemes. More than one scenario can be de ned in order to investigate the behavior of the autonomous controller under di erent conditions. We consider three general cases with which an autonomous controller shall deal: (i) nominal execution, when everything goes as expected; (ii) dynamic goal injection, an extension of the nominal execution in which one or more goals are dynamically included during the system operation; and (iii) execution failure, when some components of the system induce a not nominal behavior so as to force the controller adapting its plan to overcome the contingency. Failures in that case can be due to external perturbations, mechanical failures or degradation of the system over time.
Tests Execution. Performing tests entails the execution of each scenario that is to be monitored. Typically, uncertain and/or uncontrollable tasks are part of the problem, so, each scenario should be performed several times, to collect average behaviors and metric values.
In this regard, a scenario instantiation step is required to generate the set of models needed to de ne a suitable set of planning domains and required goals. Also, autonomous controllers can be deployed with di erent internal settings and, thus, scenarios instantiation should consider also to enable the execution of tests under di erent conditions.
Then, actual tests execution is needed. This is an important step for instantiating, executing, monitoring and collecting the data after several executions of an autonomous controller in a given scenario. During the tests execution modi cations of the nominal execution should be considered, by automatically injecting goals or failures to also test not nominal scenarios.
Report and Assessment. Once all the tests are completed, a report on the information gathered during the several executions shall be provided. Reports contains an insight of the controller behaviors, providing values for each metric as well as generating compact views, e.g., by means of graphical representations to support the users while analyzing system performances.
In fact, the information provided within reports is to inform users and enable a performance assessment allowing an objective evaluation of the control architecture in the di erent considered scenarios. After execution, a huge amount of generated data is expected and then a general representation for the data produced is to be de ned. 2.1
Metrics de nition and graphical report De nition and presentation of metrics deserve a more detailed discussion and, in the following, a detailed formalization is provided. In the above methodology, a set of metrics M is considered, being a metric denoted as i 2 M and de ned in a range lib i iub with lib and iub are (respectively) the lower and upper bounds for the i metric. Also, for each metric an extra parameter is to be considered, i.e., the weight, iW , that represents the relevance of the metric within the global evaluation. Considering the size of M (i.e., the number of de ned metrics) as n, the sum of all weights is supposed to be 100: n X i=0 iW = 100 (1)
After execution, the average value for each measured metric, iV , is considered in the report and this value is considered to compute a metric score iS as follows: iS = 100 j iub lb i j "C "T (2) considering that the upper bound of the metric is the worst score. If the metric value is out of the de ned range, its score is 0. Last factor, "C ="T , expresses the impact of execution failures in the metrics scores, being "C the number of correct executions and "T the total runs.
In order to objectively evaluate and compare autonomous controllers we need to use a common set of metrics. In this way, for the evaluation presented in this paper we have gathered the following metrics that can be measured in any autonomous controller: Operational time. Is the time spent by each part of the controller to update its internal state and schedule its goals for execution Goal processing time. The time required for each part of the controller to analyze what are its particular objectives.
State processing time. Is the time required by each part of the controller to analyze the incoming information from other part of the system, such as the sensors or other layers states required to evaluate its own status. Deliberation time. Is the time spent by the controller in generating a long-term plan to achieve its goals.
In the current metrics set presented before we are not taking into consideration
the execution time as part of the evaluation. However, it is possible to de ne
metrics in which the execution time is relevant (as a factor of the iS ).
It is worth underscoring that we are not measuring the time that the functional
layer takes to complete the actions: our methodology is focused on the
deliberation and executive capabilities of a controller. Also, some highly specialized
works are focused on analyze the functional support {i.e. [
Graphical report. As stated before, a suitable way to provide reports is by
means of graphical representations. For example, in Autonomous Levels For
Unmanned Systems (ALFUS) [
In a similar way, here, a circular graphic representation, such as the one depicted in g. 1, is proposed to represent the autonomous controller performance. Such representation presents three di erent areas. Namely, starting from the center, the Global Score (GS), the execution times and the metrics scores area.
The Global Score (GS) presented in the center of the gure represents a
synthetic evaluation for the architecture in a scale between 0 and 10, that can
be compared with the Sheridan's model [
GS =
Pn i=0
S i 1000
W i (3)
Surroundings the GS area, three circular bars are depicted. These bars represent the average time required by the considered autonomous controller to complete each scenario. Starting from the center, these bars represent: the execution time in (i) nominal execution, (ii) in dynamic goal injection and (iii) execution failures.
Finally, the external ring in the chart is decomposed into four quadrants. The smallest circumference of the ring represents the smallest score for a metric ( iS = 0, when the metric score is equal or bigger to its upper bound), while the outside circumference is the best value ( iS = 100, or a metric value closer to the lower bound). In each quadrant there are one or more metric scores represented as a lled circular sector. So, depicting a metric requires metric weight ( iW provided by the user) and metric score ( iS obtained from the execution using eq. 2). As a result, the higher is the weight of the metric and its score, the higher is the lled area of the ring. Then, a evaluation with a GS of 10 is this one in which the metrics score lls all the ring.
This methodology constitutes a generic and reproducible process to evaluate autonomous controllers while considering varying execution scenarios. In this regard, the de nition of metrics, i.e., weights, bounds and experimental cases, is the basic step on which rely to reproduce the evaluation results. Also, with the proposed minimal metrics set and graphical representation, we can analyze and compare di erent aspects of controllers performance in an straightforward manner.
Autonomous controllers are often tested using hand tailored testbenchs. Such efforts require a great work that is speci cally done for the controller and platform under study and, thus, hardly exportable and reproducible.
OGATE constitutes an engineering e ort that, taking advantage of the research e ort described above, aims to facilitate the de nition, execution and reporting of large testbenchs that can be shared and reproduced between different researchers. In this regard, OGATE is a testing environment that can be exploited in order to implement a suitable sequence of evaluation steps for supporting the objective assessment of autonomous controllers.
To achieve these objectives OGATE provides services for instantiating, executing and monitoring the required components of an autonomous controller, while generating reports after tests execution with the collected information under a uni ed and controlled environment. Furthermore, OGATE also constitutes an interactive tool to help designers and operators of autonomous controllers providing an interface for in-execution control and inspection of the controlled system during execution.
Figure 2 provides a conceptual vision of the OGATE system in which the three relevant modules that constitutes the framework are depicted. These modules are directly related to the main services provided: instantiation is responsibility of the Mission Speci cation module, while execution and monitor are carried out by the Mission Execution. Finally, the Graphical User Interface (GUI) enables the user to interact with the system in a friendly manner, trying to encapsulate the complexity of the controlled system.
When we have de ned the tests objective, metrics and the controller to evaluate, we need to provide OGATE the required information that enables the system to attach the di erent con gurations of the controller under study, the scenarios and the relevant parameters to measure. This is done by means of an eXtensible Markup Language (XML) con guration le that can be created within the OGATE GUI. The tool is general and does not provide the metrics to measure, is responsibility of the user to de ne them. In OGATE the metrics are represented by its name and the required values to perform the evaluation presented previously {at least the value range, relevance of the metric in the nal evaluation and position in the graphical report. The con guration of the autonomous controller entails some engineering knowledge of the controller, while con guring the scenarios {goals and metrics{ is more related to operators and planning experts skills. Also, the platform (real or simulated) exploited for the tests shall be properly provided.
A particular capability of the OGATE system is the possibility of automatically generate di erent scenarios and controller con gurations by exploiting a template schema. In the OGATE con guration le it is possible to de ne different parameters {i.e goals, initial conditions among others{ as templates, and, for each template, provide a set of instances. Before execution, OGATE is able to combine all instances possibilities to automatically attach the con guration les to create di erent scenarios and con gurations to be tested.
With the information provided in the XML con guration le, OGATE performs the execution by activating the di erent components of the autonomous controller, accordingly to the con gurations provided. Then, the tool is in charge of supervising and monitoring the controller execution by inspecting internal monitors of the di erent components and retrieving relevant information about its performance. When testing challenging scenarios {dynamic goal injection or execution failures{, OGATE is also responsible of modifying the nominal execution by interacting with the controlled system sending the required telecommands and/or telemetry messages that lead to include a new goal in the controlled system or to modify the execution outcomes. The telemetry/telecommands required shall be provided by the user in a format that is understood by the autonomous controller; OGATE acts as a relay by simulating the operator {goal injection{ or the functional support {execution failure.
Finally, the collected information are exploited to generate detailed reports to support assessments based on the analysis of the considered metrics. In this way, OGATE {at the end of the execution{ provides a graphical report as the one presented in g. 1, but also the temporal pro les of the selected metrics and their representative values {minimum, maximum, average and aggregated value{ in a Comma Separated Values (CSV) le that can be assessed with other analysis tools. Also, during execution, the OGATE GUI provides to the user the representative metrics values and temporal pro les in real-time.
In order to control and to retrieve data from the controlled system, some parts of the autonomous controller shall be accessible during execution. To deal with this requirement, OGATE implements a set of interfaces to enable external system interconnection. Those parts which implement interfacing with OGATE are called OGATE plugin. By means of these interfaces, the status of a plugin can be monitored and modi ed by OGATE, while also the relevant metrics can be gathered and provided to the user during execution. The implementation of such interfaces in the autonomous controller shall be done to exploit the OGATE capabilities; anyway, a similar e ort shall be done in order to perform hand tailored tests campaigns. In this sense, the bene ts of exploit OGATE are related to the saved e ort related to prepare the tests campaign and the later data collection and analysis.
Regarding this, to work with OGATE, rst it is required to perform an engineering e ort to implement the required interfaces to enable the control and data inspection of the relevant parts of the autonomous controller and, then, provide the XML con guration le, including the controller con guration, scenarios description and metrics to measure. With this information, OGATE automatically performs tests execution, data gathering and report generation at the end of the execution. Technically, OGATE is implemented in Java and the communication with the autonomous controller is done by means a simple message protocol constructed over TCP/IP.
Finally, OGATE has been designed to directly connect the planning and/or execution layers of an autonomous controller. So, performing experiments with either simulated or actual robotic platforms is not relevant 4
To assess the test campaign presented later in this paper, we have employed a planetary exploration case study employing the DALA robotic platform. In particular, DALA is an iRobot ATRV robot that provides a number of sensors and e ectors, allowing to be used for autonomous exploration experiments. It can use vision based navigation, as well as a Sick laser range nder, being the vision system formed by two cameras mounted on top of a Pan-Tilt Unit (PTU). Also, it has a panoramic camera and a communication facility
In this paper to execute tests, the DALA rover has been simulated by means
of a software environment3 based on OPRS [
The objective of the robotic platform is to address a planetary exploration problem. The mission goal is a list of required pictures to be taken in di erent locations with an associated PTU con guration. During the mission, the Ground Control Station (GCS) may be not visible for some periods. Thus, the robotic platform can communicate only when the GCS is visible. A graphical representation of the problem is presented in g. 3.
The rover must operate following some operative rules to maintain safe and
e ective con gurations. The conditions that it must hold during the overall
mission are: (C1) while the robot is moving the PTU must be in the safe position;
(C2) pictures can only be taken if the robot is still in one of the requested
locations while the PTU is pointing at the desired direction; (C3) once a picture
has been taken, the rover has to communicate the picture to the GCS; (C4) while
communicating, the rover has to be still; and (C5) while communicating, the
GCS has to be visible.
3 DALA software simulator courtesy of Felix Ingrand and Lavindra De Silva from
LAAS-CNRS.
Thanks to the Robotic Department of the European Space Agency (ESA) we
have been able to use the Goal Oriented Autonomous Controller (GOAC) [
GOAC aims to constitute a general purpose autonomous controller capable to be tailored for di erent missions/platforms. In that sense, a GOAC instance is a determined and functional con guration to successfully accomplish an objective. The aspect that determines the capabilities of the architecture is the number and hierarchy of the TREX reactors. A reactor is an entity that operates over one or more timelines by (a) deliberating over their required status to achieve the mission goals and/or (b) modifying the status of the timelines as a result of an operation or for an environment change.
In this paper, we exploit a rather simple instance with two reactors as shown in g. 4: a Deliberative reactor and a Command dispatcher reactor. The rst one is responsible of performing the deliberative task given a domain and a problem encoded in Domain De nition Language (DDL) and Problem De nition Language (PDL) respectively, following a sense-plan-act paradigm. The deliberative reactor can operate with two di erent planning policies: a single goal policy, in which goals are planned as a sequnce (i.e., one after the other), following a sort of batch schema; or, a all goals policy, in which a unique planning step generates a solution plan for all the goals. The Command dispatcher is in charge of executing commands and collecting execution feedback, being connected to the functional layer.
A plan in GOAC has the form presented in g. 5, in which the involved timelines are depicted. The Deliberative reactor generates the di erent transitions accordingly to the constraints and temporal relations de ned in the domain, while the Command dispatcher encodes the planned values into actual commands for the rover and uses the feedback provided by the functional layer to produce observations on the low-level timelines that represent the current status for the robot systems.
Finally, it is worth observing that in GOAC the planning and execution are interleaved: while the functional layer is executing a command, the executive is permanently observing the environment, so, it is capable of detect changes and respond in a short time by exploiting reactive planning schemes, instead of perform a replanning process, often more expensive. 6
This section presents the evaluation of the performance for the GOAC autonomous controllers using the planetary exploration case study with the DALA robotic platform introduced above. The evaluation takes advantage of the OGATE framework to automatically perform the tests under di erent circumstances { nominal execution, dynamic goal injection and execution failure. The experiments have been ran on a PC endowed with an Intel Core i5 CPU (2.4GHz) and 4GB RAM.
More speci cally, to perform the tests execution, a suitable GOAC plugin for OGATE has been implemented in order to send to OGATE all the relevant information from the internal components. Also, the di erent con guration parameters for the GOAC system have been adapted to exploit the OGATE template system. In this way, di erent templates have been de ned to identify the deliberative planning policy, mission goals, temporal uncertainty in action durations and the number of communication opportunities. More in particular, for the di erent templates we have provided the following set of instances varying the complexity of the problem and the execution conditions: (i) planning policy, selecting between the single goal or the all goals; (ii) plan length by increasing the number of requested pictures (from 1 to 3); (iii) plan exibility or temporal uncertainty, in which for each uncontrollable activity (i.e., robot and PTU movements as well as camera and communication tasks), a minimal duration is set, but temporal exibility on activity termination is considered, i.e., the end of each activity presents a tolerance ranging from 0 to 10 seconds; and (iv) plan choices as function of the number of communication opportunities spanning from 1 to 4. In general, among all the generated problems instances, the ones with higher number of required pictures, higher temporal exibility, and higher number of visibility windows result as the hardest.
Then, OGATE has been exploited to: (i) generate the considered scenarios, (ii) carry out all the di erent controller executions and (iii) collect performance data from the controller. For each execution setting, 10 runs have been performed and average values for the de ned metrics are reported. After the collection of performance information in all the considered scenarios, OGATE is able to generates a report containing a wide set of charts corresponding to di erent control con gurations, planning problem instances and execution settings. Finally, for the 10 executions we have considered 4 nominal executions, 3 with goal injection and 3 for execution failure. For the dynamic injection scenario, a new picture is requested between the seconds 40 and 60 after the mission begin and, for the execution failure, a miss-con guration of the PTU occurs during its rst reorientation.
For measuring performance, we exploited the metrics presented earlier in
the paper. The metrics are captured for both reactors in the GOAC controller
and the following ranges (in seconds) have been considered for the one picture
scenario: [
Considering all the possible combinations, we obtain 72 possible instances of the GOAC controller. For each of them we obtained a graphical report such as the one presented in g. 1 { that particular one shows the scenario for one picture with the single goal policy, 4 communications opportunities and 10 seconds of temporal exibility. As we cannot provide a detailed discussion on each possible scenario, we will focus on the Global Scores computed for each instance, as stated in table 1.
A rst straightforward evidence that can be elicited observing the Global Score is that the controller performs similarly with 1 and 2 pictures for both planning policies but has a performance fall for the all goals when executing scenarios with 3 pictures that does not occur with the single goal.
In all the tested scenarios, the GOAC deliberative component is able to generate a valid plan, but the controller fails in properly completing its execution in some of them. In particular, the execution failure scenario is never completed: when the deliberative component receives the PTU miss-con guration, it does not correspond to its planned states, producing a failure that leads to a system halt. Otherwise, the nominal and dynamic goal injection scenarios are usually completed, except in particular cases of the all goals policy : for 1 picture with 2 and 3 communications opportunities and 10 seconds of temporal exibility; for 2 pictures with 2 communications opportunities and 10 seconds of temporal exibility; and, for 3 pictures, the all goals has several problems to achieve the mission goals except for the one communication opportunity scenario, in which completes the nominal and dynamic goal injection scenarios for all temporal exibilities. Instead, the single goal policy only fails to perform the dynamic goal injection scenario for the hardest cases: 3 pictures with 3 and 4 communications opportunities and 10 seconds of temporal exibility. If we analyze the average values for the di erent planning policies clustering only the scenarios by the number of pictures, we can see that there is no relevant di erence between both planning policies for 1 and 2 pictures, but, for 3 pictures, the single goal policy seems to be more adequate to be deployed. In fact, the single goal policy usually outscores the all goals policy. 7
This paper has presented some recent results in addressing the open issue of evaluating the performance of a planning and execution system. To deal with this problem the paper rst proposes a methodology to properly guide the testing phase and achieve an objective evaluation. Second, it describes the operationalization of such methodology in a software environment, called OGATE, that is able to perform large test campaigns for di erent challenging scenarios without user interaction.
The described approach has been used to test the GOAC autonomous controllers. The paper has presented a minimum set of metrics over which the controller performance has been pro led. It is worth underscoring that performing such tests without the described tool requires a large amount of time and a non trivial work to set up di erent con gurations and retrieve information related to the considered metrics. The experiments have been able to characterize the di erent planning policies of the GOAC deliberative component and the performance of the system as a function of the complexity of the given problem.
Among future works, the de nition of a more thorough set of standard metrics constitutes a key immediate step. Additionally, OGATE will be used to compare di erent plan-based deliberative platforms on the same benchmark tests (a natural extension of the current status).
Pablo Mun~oz is supported by the European Space Agency (ESA) under the Networking and Partnering Initiative (NPI) Cooperative systems for autonomous exploration missions. CNR authors are partially supported by the Italian Ministry for University and Research (MIUR) and CNR under the GECKO Project (Progetto Bandiera \La Fabbrica del Futuro"). Authors want to thank to the ESA's technical o cer Mr. Michel Van Winnendael for his continuous support.