In this position paper, we claim that the availability of quantitative predictions of performance of robots in unseen environments can significantly contribute to the progress of autonomous robotics. Moreover, we overview a methodology that we adopted to provide such predictions in the case of Simultaneous
Imagine a scenario in which a search and rescue human team has to deploy an autonomous robot system during a rescue operation in a large site, like an industrial plant, and has to choose, between diferent configurations of the robot system, the one that maximizes the eficiency in finding possible victims. Consider another scenario in which a company that develops autonomous robot systems for assistance to elderly people has to configure a new installation in a newly built residence for retired people in order to make the robots moving as safely and as robustly as possible. Finally, consider a third scenario in which a consumer has just bought a new autonomous cleaning robot for the ofice that he/she manages and has to manually set the best configuration of parameter values. Currently, to address all the above scenarios (and several others) and make the decisions they involve, autonomous robots developers and users mostly rely on trial-and-error approaches and on subjective previous experience. Trial-and-error approaches can be significantly time consuming and expensive. Subjective previous experience is something that could be expected from the experts (like the search and rescue human team) but not from end consumers.
One fundamental problem that characterizes the above settings is the dificulty in
predicting
the performance of a robot system in a new scenario. Indeed, beyond informal experience of
researchers and developers, there is not any assessed methodology to quantitatively estimate
the expected performance of a given robot system in a given scenario before actually running
the system in the scenario. Methods for performance evaluation work almost exclusively ex
0A preliminary version of this paper appeared as [
Artificial Intelligence and Robotics (AIRO) Workshop at AIxIA 2021 nEvelop-O post, namely after the actual operations of the robots.
In this position paper, we argue that the development of innovative methodologies for
predicting the performance of autonomous robots in new scenarios could represent a major
contribution to the advancement of the field. In a sense, these methodologies will enable a sort
of predictive benchmarking of autonomous robot systems, meaning that one could be able to
anticipate, up to some degree of confidence, the performance results obtained by a system on a
benchmark. There have been some attempts, especially in the military field, to predict robot
performance in previously unseen scenarios. An example is [
Methodologies to quantitatively predict robot performance are expected to be highly significant for at least three aspects.
First, the availability of reliable predictions on performance of autonomous robot systems can help developers to make informed decisions at design time, reducing expensive trial-and-error iterations. Consider the following scenario. A urban search and rescue team has to deploy a multirobot system in a building in which a gas leakage occurred in order to assess the presence and the location of possible human victims. The multirobot system is composed of several communicating mobile robots that should explore the initially unknown building and report to the search and rescue team a map of the site with the locations of the victims, so that the human rescuers can enter the building and quickly reach the victims, reducing risks for their safety. Assume that the multirobot system can have two diferent configurations A and B. In configuration A, the robots are equipped with powerful laser range finders (e.g., with field of view of 360 degrees and range of 70 m), have an extended communication range (e.g., 100 m without obstacles), but are slow (e.g., they cannot move faster than 0.5 m/s). In configuration B, the robots are equipped with less powerful laser range finders (e.g., with field of view of 200 degrees and range of 15 m), have a limited communication range (e.g., up 15 m without obstacles), but can move faster (e.g., they can move at 2 m/s). Currently, there is no principled way for the search and rescue team to choose a priori (before actually deploying the robots) the best configuration between A and B, beyond informally relying on past experience. Methodologies to predict robot performance will allow the team to make an informed decision about which configuration is better to use. This kind of predictive benchmarking perfectly fits with the Ethically Aligned Design approach of the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems1.
Second, knowledge of environments in which a given autonomous robot system is expected to
perform well and in which it is expected to perform bad can be used to establish the boundaries
of its applicability. This can enable the generalization of results obtained by robot systems [
Finally, the social impact of the methodologies to predict robot performance can be in the direction of an improved social trust in autonomous robots. Quantitative assessments of the performance of robot systems that can be reliably generalized can help to improve the degree of confidence and acceptance of such robot systems. This concern is addressed, among others, by the Foundation for Responsible Robotics2. For example, consumers will be more inclined to trust an autonomous robot system for domestic assistance if its performance measured in other houses quantitatively suggests a good level of performance also in their houses.
We started to develop a methodology for quantitatively predicting the performance of a
Simultaneous Localization And Mapping (SLAM) [
We consider a SLAM system, denoted with , composed of a mobile robot platform equipped with sensors for detecting the surrounding environment (a laser range scanner with a field of view of 270∘, an angular resolution of 0.5∘, and a maximum range of 30 m in our case) that is running a SLAM algorithm (GMapping [7] in our case). Given a SLAM system , the methodology consists of a sequence of steps.
At first, we perform data collection. Specifically, given a set of environments and , we collect the performance (, ) of in each environment ∈ . Performance () is calculated as the localization error according to [8], which requires to know the actual trajectory followed by the robot (ground truth). Note that, since performance varies over diferent runs of the same robot system in the same environment (e.g., due to noise in movements and perceptions), (, ) involves multiple runs, which we perform in simulation and with real robots.
Then, for each environment ∈ , we extract a set of features () which characterize . In this step, called feature extraction, we obtain the features from knowledge available a priori (e.g., obtainable from floor plans of buildings) and not from the maps built by the robots. We consider geometrical features (like area and perimeter of ) and structural features (like those derived from the Voronoi skeleton of [9]).
The third step is model learning, in which a model that relates () to (, ) (for any ∈ ) is built. In general, a model correlates a subset of features of () to (, ) . In our case, we found a strong correlation (using linear regression) between the length of an exploration path over the Voronoi skeleton of and the performance () . Next section discusses numerical results.
Such model is used to do performance prediction, namely to calculate the expected performance (,̂ ) of in an unseen environment (namely, ∉ ). Our performance prediction extracts features () from and finds (,̂ ) exploiting the model built in the previous step.
In principle, the above sequence of steps can be repeated when the set of visited environments grows, resulting in increasingly accurate performance predictions.
The proposed methodology is based on two main assumptions: (a) models relating () (b) a set of features () environment .
to (, ) for all environments ∈ can be found and can be obtained without actually deploying the robot in the new We practically validated assumption (a) in the case of GMapping and in the case of another SLAM system based on KartoSLAM [10] (results are not discussed here). Hence, we are confident that similar models, perhaps not linear and possibly based on other techniques (like neural networks), can be found also for other SLAM systems and, more generally, for other robot systems. Note that, if a large amount of data is available, feature extraction and model learning steps can both be addressed using learning techniques like deep neural networks, reducing the human efort required in adapting the proposed method to new applications. Assumption (b) basically implies that some knowledge about is available before robots are actually deployed in . For example, such knowledge could consist in the floor plan of (e.g., obtained from urban administrative ofices or from evacuation maps) from which () can be extracted. Note that knowing the floor plan of does not amount to knowing the map of , because unforeseen obstacles that might occupy the free space, like, for instance, furniture, are not included in the lfoor plan.
In this section we present a sample of results we obtained with the proposed approach.
We collect data on performance of GMapping on a set of 100 indoor environments covering a wide range of building types (schools, ofices, university campuses, and others), sizes, and shapes (to avoid overfitting). Environments are taken from those used in [ 11], a subset of 11 of the 20 environments of [11], a set of 25 floor plans that represent buildings of the MIT university campus from [12], and our own dataset of 64 floor plans representing real world buildings [ 13]. Simulations are performed with Stage, using the ROS GMapping3 and Navigation packages4 for SLAM and robot movement, respectively. Given an environment and a starting pose for the robot (close to the center of the environment, the same for all runs), a run evolves by exploring using the frontier-based exploration paradigm defined in [ 14]. Full details about data collection are reported in [15].
We extract a set of features () that characterize an environment using the Voronoi graph built on [16]. Intuitively, the edges of the Voronoi graph represent the clearest paths (i.e., the paths furthest from obstacles) that a robot could follow in the environment. We exploit the Voronoi graph to obtain an approximation of the path a robot follows to build the complete map of an initially unknown environment, considering the limitations of the robot sensors. In 3http://wiki.ros.org/gmapping 4http://wiki.ros.org/navigation practice, given the floor plan of , represented as a grid map (image), we extract the Voronoi graph from the dual Delaunay triangulation, as specified in [ 9].
Given the Voronoi graph of an environment , we calculate two quantities, the Voronoi traversal distance (VTD ) and Voronoi traversal rotation (VTR ), that are the overall distance and the overall amount of rotation, respectively, that a robot incurs to cover the Voronoi graph of .
We build a model of the relationships between performance (, ) (localization error, calculated as in [8]) and features () for all the environments ∈ . To assess the quality of a possible model, we compute the overall root mean square error (RMSE) and the average coeficient of determination ( 2) on the training set using k-fold cross validation (with a number of folds equal to 5). We also consider the normalized (scale-independent) version of the RMSE: NRMSE = RMSE , where max and min are the maximum and the minimum observed max− min values of the target variable , respectively.
Table 1 shows the best model, in terms of highest 2 value for the localization error we are
interested in predicting, and the corresponding results, which show a good performance of our
approach. For more details, please refer to [
In conclusion, we think that the methodology depicted above can be generalized to different robot systems and to diferent performance measures , enabling the possibility of getting quantitative predictions of the expected performance of robot system in new, unseen, environments. [7] G. Grisetti, C. Stachniss, W. Burgard, Improved techniques for grid mapping with Rao
Blackwellized particle filters, IEEE T Robot 23 (2007) 34–46. [8] R. Kümmerle, B. Steder, C. Dornhege, M. Ruhnke, G. Grisetti, C. Stachniss, A. Kleiner, On measuring the accuracy of SLAM algorithms, Auton Robot 27 (2009) 387–407. [9] A. Bowyer, Computing Dirichlet tessellations, Comput J 24 (1981) 162–166. [10] K. Konolige, G. Grisetti, R. Kümmerle, W. Burgard, B. Limketkai, R. Vincent, Eficient sparse pose adjustment for 2D mapping, in: Proc. IROS, 2010, pp. 22–29. [11] R. Bormann, F. Jordan, W. Li, J. Hampp, M. Hägele, Room segmentation: Survey, implementation, and analysis, in: Proc. ICRA, 2016, pp. 1019–1026. [12] E. Whiting, J. Battat, S. Teller, Generating a topological model of multi-building environments from floorplans, in: Proc. CAADFutures, 2007, pp. 115–128. [13] M. Luperto, A. Quattrini Li, F. Amigoni, A system for building semantic maps of indoor environments exploiting the concept of building typology, in: Proc. RoboCup, 2013, pp. 504–515. [14] B. Yamauchi, A frontier-based approach for autonomous exploration, in: Proc. CIRA, 1997, pp. 146–151. [15] F. Amigoni, V. Castelli, M. Luperto, Improving repeatability of experiments by automatic evaluation of SLAM algorithms, in: Proc. IROS, 2018, pp. 7237–7243. [16] S. Schwertfeger, A. Birk, Map evaluation using matched topology graphs, Auton Robot 40 (2016) 761–787.