ARIAC (Agile Robotics for Industrial Automation Competition) is a robotic competition which aims to advance robotic agility in industry. Participants in this competition are required to implement a robot control system to overcome agility challenges in a simulated environment. ARIAC comes with a set of score metrics to evaluate the performance of each control system during task execution. In this paper we show how such task-oriented evaluation can be problematic and how the addition of runtime monitors to verify properties given in ISO/TS safety standards can help in reducing the resulting reality gap.
Organised by the National Institute of Standards and Technology (NIST, https://www.nist.gov/) since
2017, the Agile Robotics for Industrial Applications Competition [
The competition participants are required to develop a robot control system for a gantry robot in order to perform kitting in a simulated environment. Gazebo (http://gazebosim.org/), which is an open source robotics simulation environment, is used as the testing platform and the Robot Operating System (https://www.ros.org/) (ROS), which is an open source set of software libraries and tools, is used to define the interfaces to the simulation system.
Figure 1 depicts the simulated environment where the ARIAC competition takes place. The robot used in ARIAC 2020 is of gantry type and can move in the simulated environment to interact with objects in order to perform kitting tasks. A kit is an order for specific items, which can be found on shelves, on the conveyor belt, and in bins. The robot builds kits by picking up all the required items and placing them into one of the two trays located on the Automated Guided Vehicles (AGVs). When an order is completed, the AGV delivers the kit and a final score is given to the participants’ systems. The final score takes into account many aspects, such as the type/color of the selected item matches the type/color required by the order; the accuracy of products’ pose in the tray; and the time taken by the control system to complete a kit (measured in simulation seconds).
Other metrics are used to determine the quality and performance of each participant’s robot controller. However, except for some limited and manually coded control, there is no evaluation of how the robot completed the orders. The score considers if the right item has been brought to the delivery station, but does not take into account how the robot actually builds the kits. For instance, during kitting, the robot may get very close to people in the environment or even hit them, as seen in Figure 2. Safety distance was not monitored in ARIAC 2020 and collisions with humans could only be reported by ARIAC organisers after the fact. A participant’s control system may put the robot in these unsafe situations and still get the highest score. As can be seen, a balance between performance and safety of a robotic system has to be implemented in ARIAC. Above all, there is a need to reduce the reality gap between such implementations and the real world.
One way to solve this problem is to give importance to such safety aspects while evaluating a robot controller. To obtain this, we added runtime monitors to check the robot controller’s behaviour at runtime. A runtime monitor is a component which observes a running system, and checks if the observed behaviour satisfies the expected behaviour through formal properties. The resulting verification process is called Runtime Verification (RV). With respect to other verification techniques, RV has the advantage of focusing on the current system execution, rather than statically trying to generate all possible ones; this makes it a lightweight approach which can be even applied to large and complex scenarios.
The main reason we chose RV is that it can be applied to black-box scenarios. Since the monitor only needs to observe how the robot behaves in the simulated environment, it does not need to know how the robot takes its decisions under the hood. Consequently, participants’ source code is not required and the monitors are totally independent from the robot controller’s implementation.
As a proof of concept, we developed a simple monitor to verify that the robot satisfies the safety
distance from the simulated humans. ISO/TS 15066:2016 - “Robots and robotic devices — Collaborative
robots" addresses the safety issue of robot speed and separation monitoring [
“the distance between the robot and the human operator is always greater than or equal to ”
Since ARIAC is implemented in ROS, we used ROSMonitoring1 [
As mentioned previously, the monitor observes the system while the latter is running. Since the system is implemented in ROS, the monitor observes the messages exchanged among the nodes2 in the system. The monitor is automatically synthesised by ROSMonitoring, but it requires knowing which messages have to be observed and which property has to be verified.
1https://github.com/autonomy-and-verification-uol/ROSMonitoring 2ROS is node-based; a robot can be composed of diferent nodes, and the nodes communicate through message passing.
Regarding the messages, in ARIAC there was no explicit representation of robot and human speed, or their distance. More specifically, this information was available in the simulation, but there were no explicit messages exchanged. Since ROSMonitoring monitors intercept messages, messages related to this information were added. A ROS node was created to keep track of where the robot and the human operators are, and at what speed they are moving. This node then publishes this information at a custom rate, which has been fixed to 100 Hz in this specific scenario (100 messages containing information about the robot and human operators are published per second). The resulting message represents a snapshot of the system, and can be intercepted by a monitor to verify the satisfaction of properties.
Previously, we reported the property of our interest in natural language; since ROSMonitoring
natively supports RML (Runtime Monitoring Language3) [
The resulting monitor observes events characterising snapshots of the system, and verifies that each event satisfies the distance constrained derived by ISO/TS 15066:2016. If an event fails such a constraint, which means it does not belong to the event set denoted by gteq_dmin, an error is raised by the monitor, since an inconsistent event has been observed. In Figure 2 we reported a screenshot obtained with a sample robot controller. Figure 2 shows a situation where the robot is about to hit a human operator; in this situation, the monitor detects this violation of ISO/TS 15066:2016, and throws and logs an error.
We have briefly presented ARIAC, and how it is used for advancing robotic technologies in the context of agility. We recognised a limitation in evaluating robotic controllers by only focusing on task completion, and how such limitation causes a reality gap. We showed how such reality gap can be reduced by adding runtime monitors verifying the system behaviour at runtime. In particular, we focused on an initial case study where a safety property given in ISO/TS 15066:2016 is used to synthesise a monitor. For future work, we aim to synthesise additional monitors deriving from other relevant standards, and integrate the resulting RV process in the score evaluation of future ARIAC competitions.
3RML is more expressive than LTL [