Intelligence Level Performance Standards Research for Autonomous Vehicles

Abstract

- United States and European safety standards have evolved to protect workers near Automatic Guided Vehicles (AGV's). However, performance standards for AGV's and mobile robots have only recently begun development. Lessons can be learned from research and standards efforts for mobile robots applied to emergency response and military applications. Research challenges, tests and evaluations, and programs to develop higher intelligence levels for vehicles can also used to guide industrial AGV developments towards more adaptable and intelligent systems. These other efforts also provide useful standards development criteria for AGV performance test methods. Current standards areas being considered for AGVs are for docking, navigation, obstacle avoidance, and the ground truth systems that measure performance. This paper provides a look to the future with standards developments in both the performance of vehicles and the dynamic perception systems that measure intelligent vehicle performance.

-

I. INTRODUCTION

Automatic Guided Vehicles (AGV’s) have typically been used for industrial material handling since the 1950’s. Since then, U.S. [ 1 ] and European [ 2 ] AGV safety standards have evolved to protect nearby workers. These standards have minimal test methods to describe how manufacturers and users are to perform AGV safety measurements, resulting in potential measurement differences across the industry. For example, American National Standards Institute/Industrial Truck Safety Development Foundation (ANSI/ITSDF) B56.5:2012 provides new language to generically handle a situation when an object suddenly appears within the AGV stop region. The stop region is the area surrounding the AGV in which the non-contact safety sensor detects obstacles and stops the vehicle. The manufacturer must now prove that when the AGV detects an object closer than its stopping distance, although collision with the object is perhaps imminent, the AGV demonstrates a reduction in kinetic energy. However, there is no description of how manufacturers measure this situation, resulting in different measurement results across manufacturers. One test method was researched to handle this situation and is described in [ 3 ].

Recently AGV and mobile robot performance standards developments have begun to limit measurement method differences. Initial developments began with a review of other research and standards efforts for mobile robots as applied to emergency response and military applications [ 4 ]. This reference also discusses research challenges, test and evaluations, and intelligent systems development programs that can support advancement of industrial AGVs towards attaining greater levels of intelligence. These other efforts also provide useful standards development criteria for AGV performance test methods. Experiences and results in advanced mobility and intelligence for robotics will be essential for AGV manufacturers and users to fully understand capabilities and specific applications of their autonomous vehicle systems.

Performance test methods for docking, navigation, (see Figure 1) [ 5 ], and terminology standard work items have been initiated under the new ASTM Committee F45 on Driverless Automatic Guided Industrial Vehicles performance standard [ 6 ]. Standards for autonomous industrial vehicle obstacle avoidance and protection, based on past research [ 7 ], communication and integration, and environmental impacts are also being considered.

This paper will specifically discuss measurement of: vehicle navigation (e.g., commanded vs. actual AGV pathfollowing deviation), vehicle docking (e.g., AGV stop point positioning vs. known facility points), and obstacle detection and avoidance of standard test pieces (e.g., comparison of realtime AGV path-planning and new path following vs. commanded path) towards smart manufacturing applications, such as assembly and unstructured environment navigation. Additionally, this paper will discuss a new ASTM Committee on 3D Imaging Systems E57.02 [ 8 ] standard work item for six degree-of-freedom (DOF) optical measurement of dynamic systems (see Figure 2), which advances the existing static 6 DOF standard [ 9 ]. The new standard is expected to be a critical component of performance measurement for current and future robotic systems that rely on advanced perception systems.

II. PERFORMANCE STANDARDS THRUSTS

AGV navigation, docking, and obstacle detection and avoidance tests were conducted in support of future performance standard test methods and are described in this section. In some instances, typical industry practices were evaluated as well as the improved AGV performance tests.

A. Vehicle Navigation

The most basic functions of mobile robots and AGV’s are navigation to and docking with equipment in the workspace. However, the description of how well the vehicle navigates (i.e., commanded vs. actual AGV path-following deviation) has certain ambiguities. For example, navigation implies that the vehicle measures its current position, plans a route to another location, and moves from the current location to planned location upon command. Most vehicle manufacturers don’t provide specifications for how uncertain the navigation performance is (i.e., the error bounds on position or velocity), other than perhaps radius of vehicle turns, maximum velocity, and maximum acceleration. The vehicle velocity sets limits on the allowable turn radius for particular vehicles. Some controllers [ 10 ], if not all, will not allow high velocities on relatively small radii to prevent unsafe vehicle conditions. These limitations are not typically specified by AGV manufactures, causing AGV users difficulty in planning how many vehicles they may require for moving their products within the facility to maintain a desired throughput.

Industrial vehicles may eventually become uncalibrated through regular use. An uncalibrated vehicle does not follow a commanded path or stop/dock at a commanded point with minimal relative uncertainty (standard deviation of measured vs. ground truth) as does a calibrated vehicle. To correct this, vehicle manufacturers have calibration procedures for their vehicles, although these procedures can be tedious, timeconsuming, and may not be appropriate for all vehicles. For example, calibration of Ackerman steered vs. ‘crab’ steered (sometimes called quad) vehicles have different calibration procedures. It is not always clear what will happen when a vehicle is uncalibrated nor when the vehicle becomes uncalibrated. The effects of calibration on vehicle control and uncertainty are typically not specified either. There is also typically no specification describing how far from the commanded path a vehicle navigates. This may be important to users who have tight tolerance AGV paths (e.g., paths between infrastructure) that must be followed. A test can be developed to uncover the effects of uncalibrated vs. calibrated vehicle navigation performance when commanded to move along a path, as shown as a dashed line in the example in Figure 1. Should objects be near the vehicle path, such as walls or obstacles, depicted in Figure 1 as bordering lines along the path, the vehicle may stop, slow, or worse, collide with the boundary object. A user would then be required to provide additional, perhaps unnecessary space for one manufacturers’ vehicle and not for another. How the vehicle handles (slow, stop, etc.) the event is also ambiguous. For example, some, but not all vehicles are equipped with obstacle detection based on non-contacting sensors that provide detection beyond the physical vehicle footprint.

To address AGV navigation uncertainty, with an eye towards a potential test method for all automatic industrial vehicles, tests were executed, both with an AGV prior to and after being calibrated. The uncalibrated AGV test is similar to typical industry methods since not all AGVs can be frequently calibrated. An uncalibrated AGV was moved along a straight line path between two commanded points in an open area and spaced approximately 5 m apart [ 5 ]. Figure 2 shows the results amplified in the X direction 100 times to exaggerate vehicle performance. In the figure, the blue line is the commanded path between points 1 and 2. The green dots to the right and left of the line are uncalibrated AGV controller-traced position data moving forward and reverse, respectively, between the points. The red dots are ground truth of the navigating AGV between points using an optical tracking system. This experiment demonstrated one AGV navigation performance measurement method using a precision (0.2 mm standard deviation) six degree-of-freedom (DOF), optical measurement system as a ground truth comparison to the onboard vehicle tracking system. Path deviation was approximately 20 cm maximum. The AGV was then calibrated using the manufacturer’s method.

Pt 1

Pt 2 Page 49

Another test setup was tried, with an eye towards a relatively less expensive test method that will allow all AGV systems to be measured, ideally, with an independent measurement method that doesn’t use AGV controller tracking, yet captures the full AGV configuration (i.e., including safety sensing). The AGV was commanded to drive back and forth between temporary barriers, along a straight line defined by commanded points spaced approximately 10 m apart. The goal of the experiment was to measure the AGV deviation from the commanded path. A critical AGV navigation performance area is also deviation from the commanded path after turns so a 90° turn was added to the end of the straight path beyond the barriers to measure the vehicle navigation uncertainty when moving from/to a straight path to/from a turn. Figure 3 shows the test setup and Figure 4 shows (a) a B56.5 test piece being used to define the safety laser stop field edges, (b) the barriers and lines to which barriers are moved between trials, and (c) the AGV emergency-stopped upon detection of the barriers. The safety laser, stop field edges were marked on the floor, as a ground truth, zero-tolerance spacing that the vehicle can navigate, when the vehicle was at position 1 and again at position 3, shown in Figure 3, for both left and right vehicle sides. The barrier position lines were measured from the edge line using a ruler and marked at 2 cm increments from the edge up to 10 cm away from the edge line. Smaller spacing between lines (e.g., 1 cm) could also be used for finer uncertainty measurement. For each test trial, the barriers were moved towards the AGV to the next line beginning at 10 cm for trial 1, 8 cm for trial 2, and so forth until the navigating vehicle detected a barrier, and emergency-stopped the AGV, thus completing the test run.

A series of eight trials were completed with nearly all trials including three or more runs each to demonstrate the navigation test method concept. Ten or more runs are ideal for statistical analysis. The optical measurement system mentioned earlier was used as an experimental ground truth (GT) to measure the barrier and vehicle position during experiments to further understand the test method and vehicle performance. The barriers and AGV were marked with spherical reflectors (visible in Figure 4 (a, b, and c) detectable from the GT system. Figure 5 presents GT data plotted for navigation tests showing ground truth data of: (a) test 8 vehicle path and emergency stopped vehicle (red circle) when a wall was detected, (b) test 1 path, and (c) test 1 path data from (b) zoomed in to show data points of three runs. blue barrierposition lines a b c

Experimental results from the barriers demonstrated a path uncertainty of between 6 cm and 8 cm maximum when the vehicle detected the boundaries at nearly the center of the straight line path and when moving at either 0.25 m/s or 0.50 m/s. The navigation test method using barriers is simple and cost-effective for manufacturers and users to employ, as compared to the higher accuracy, but more expensive ground truth visual tracking system used for test method development. A simple straight line with one turn was tested. However, more complex test configurations, such as shown in Figure 1, could be set up using B56.5 test pieces instead of larger, physical barriers as were used in this research. Page 50 when a wall was detected, (b) test 1 path, and (c) test 1 path data from (b) zoomed in to show (red, green and blue) data points from three runs.

A working document that addresses quantifying vehicle navigation uncertainty is being developed as an initial step towards a performance standard for ASTM F45.02 subcommittee on Docking and Navigation. Based on consensus of the task group developing this standard, as was tested at NIST, the simple path-bounding test method using temporary reconfigurable barriers made from readilyavailable, off-the-shelf materials is being proposed.

B. Vehicle Docking

Vehicle docking is another common application of mobile robots and AGVs. Unit load (tray, pallet, or cabinet carrying), tugger (cart pulling), and fork/clamp (pallet or box load/unloading) are typical industrial style vehicles that require different docking uncertainties. For example, a unit load vehicle that places/retrieves platters during wafer manufacturing would no doubt require less uncertainty than a fork style vehicle that places/retrieves pallets. As robotics advances, current and potential users are requesting mobile manipulators to perform tasks such as unloading trucks. Eventually, it is expected that mobile manipulators will be used for smart manufacturing assembly applications [ 11, 12 ].

Similar to navigation, there are no performance measurement test methods that define how manufacturers and users characterize their vehicle’s docking capabilities. Figure 6 (a) shows an example method for docking for any style vehicle. A vehicle approaches and makes contact with ‘a’ and/or ‘b’ docking points dependent upon the vehicle type. Relative displacement from each of the points would be measured to determine vehicle docking uncertainty. A forktype AGV is shown docked with a test apparatus in Figure 6 (b). The fork tips are marked with yellow points.

(a) (b)

Two experiments were simultaneously performed: AGV docking relative to known facility locations and GT system use for measuring AGV docking. Two different GT measurement systems were used to measure AGV performance: a laser tracking GT with an uncertainty of approximately 10 µm [ 13 ] and an optical tracking system with uncertainty of 0.2 mm in position uncertainty and 0.13° in angle uncertainty as measured at NIST. The laser tracker tracks position of a single point, whereas the visual tracking system can track multiple point markers and can computer orientation from them. Both GT systems can measure relatively high-precision displacement between two points, as compared to an AGV docking.

An experiment using an uncalibrated AGV that was programmed to stop at various points yielded an uncertainty range of approximately 1 mm to 50 mm. Figure 7 (a) shows the vehicle paths and Figure 7 (b) shows average errors for five runs at stop or dock points. The vehicle position was measured using a laser tracking GT system which provided highprecision measurement of AGV stop points. [ 13 ] However, in several experiments, laser tracker positioning was critical as the laser beam was continuously interrupted by onboard AGV hardware. This prompted a switch to using an optical tracking system for GT measurements.

A 6 DoF optical tracking GT system was used instead to measure AGV docking. Docking was measured again after the AGV was calibrated using the manufacturer’s procedures. The AGV approached similar dock locations and after AGV calibration, provided consistent 5 mm uncertainty. Standards development for optical tracking systems is also underway and is discussed in section 2 D, 6 DOF Optical Measurement of Dynamic Systems. Docking points

(b)

Additional AGV equipment docking experiments were also performed using a mobile manipulator and a reconfigurable mobile manipulator artifact (RMMA) developed at NIST (see Figure 8). [ 14 ] The mobile manipulator, with uncalibrated AGV, repeatedly moved next to the artifact from a starting point. Although uncalibrated, the Page 51 AGV provided relatively low repeatability uncertainty (e.g., +/-5 mm) although more than 10 mm from the commanded docking points. This manipulator could reach the commanded points on the RMMA even with 10 mm uncertainty in AGV position. The mobile manipulator corrected for the position uncertainty after being taught the actual RMMA locations. At the RMMA, the manipulator, wielding a laser retroreflector, was commanded to move in a spiral pattern to detect 6 mm diameter reflectors. The reflectors provide non-contact alignment detection of the tool point position and orientation. The experiment provided results demonstrating that this relatively inexpensive ground truth measurement method was sufficient for measuring docking accuracy. As the reflector based measurement system is inexpensive compared to the optical tracking-based GT, it may prove ideal for use as a precision vehicle/mobile manipulator docking test method that both manufacturers and users can replicate.

Manipulator RMMA AGV

C. Obstacle Detection and Avoidance

Obstacle detection and avoidance (ODA) research is well documented in the literature for mobile robots. However, there are few citations for AGVs perhaps due to the relatively closed nature of commercially available AGV controllers and because ODA is not often implemented on AGVs deployed in large manufacturing facilities. In [ 5 ], it was discussed that for large facilities, ODA could occur in ‘buffer zones’ (i.e., zones where AGVs would be allowed to pass other vehicles). For small and medium manufacturing facilities, however, ODA may be necessary due to more limited floor space and lesscontrolled environments. NIST has developed an algorithm, detailed in [ 5 ], and measured the performance of an AGV with added ODA capability. The algorithm is also suitable for navigating an unstructured environment although it is currently limited by the use of facility-mounted (sensors not mounted on the AGV) obstacle detection with obstacle avoidance adapted to an AGV with a controller with limited ability to integrate external algorithms. Figure 9 shows a snapshot of the ODA algorithm planning a path through multiple obstacles.

The navigation performance measurement experiment discussed previously in section II A. Vehicle Navigation can be similarly applied for obstacle detection and avoidance. In fact, the ASTM F45.02 subcommittee navigation and docking task groups have discussed the potentially overlapping nature of the two vehicle capabilities. The ASTM F45.03 Obstacle Detection and Protection subcommittee is currently in the process of considering standards in this area. Questions have been raised regarding standards development as follows: 1. 2.

How well does the AGV react to situations? For example:  Obstacles appearing in the path  Potential obstacles headed towards the path  Unstructured (i.e., changing obstacle locations) areas not on the original planned path or that rapidly change How far off the commanded navigation path can an AGV be, and at what speeds, before it violates the path and causes a stop? For example, due to environmental factors such as:  Offset-pitched/rolled AGV can’t see guidance markers, such as reflectors, magnets, wire, etc.  Guidance or boundary-marking tape is worn or broken  Terrain causes “bouncing” or moving laser or other navigation sensors How well does the vehicle react when a human is detected and how should the human be represented? For example:  By test pieces, mannequins, humans  With what coverings? (i.e., what clothes should be worn?) How to interact with manual equipment (e.g., forklifts, machines) How to standardize communication of vehicle intelligence for obstacle detection and avoidance? For example:  Contextual autonomy levels [ 4 ]  Situation awareness (e.g. LASSO) [ 14 ]:

Experiments to support ODA performance test method development will be performed based on forthcoming guidance from the ASTM F45 subcommittee. However, a prototype safety test method that has been developed to evaluate a vehicle’s response to obstacles in its path and within its stop zone, as noted in the Introduction, can be considered a first step towards full ODA standard test methods. ASTM F45 is meant to dovetail with safety standards such as ANSI/ITSDF B56.5. Therefore, providing an initial test Page 52 method for detection of obstacles is ideal as a starting point for F45.03. The ‘Grid-Video’ detection method [ 3 ] provides a simple-to-implement test method that measures positional accuracy of the dynamic test piece relative to the vehicle position when the obstacle enters the vehicle path.

D. 6 DOF Optical Measurement of Dynamic Systems

ASTM’s draft Standard for the Performance of Optical Tracking Systems that Measure Static and Dynamic Six Degrees of Freedom (6DOF) Pose (see Figure 10) is the next step beyond the static case covered by ASTM E2919-14 [ 8 ]. Optical tracking is being used for robot and autonomous vehicle GT measurement, as discussed in this paper. Optical tracking measurement systems [ 15 ] are used in a wide range of fields, including video gaming, filming, neuroscience, biomechanics, flight/medical/industrial training, simulation, and robotics. ASTM WK49831 is a working document that is considering both static and dynamic measurements of systems under test. The scope of the draft standard test method is to provide metrics and procedures to determine the performance of a rigid object tracking system in measuring the dynamic pose (position and orientation) of an object. Optical measurement systems may use the test method to establish the performance for their 6 DOF rigid body tracking pose measurement systems. The test method will also provide a uniform way to report the statistical errors and the pose measurement capability of the system, making it possible to compare the performance of different systems. So all the measurements can be traced to the standard.

Ground truth cameras AGV

In the initial test procedure, measurements with uncertainties were computed using an artifact – namely a metrology bar as shown in Figure 9 (a). Current optical tracking systems utilize a three-marker metrology bar with all markers in a line which does not provide 6 DOF system performance measurement. A metrology bar made of carbon fiber with length 620 mm and with five reflective markers attached on each end was used as the 6 DOF artifact. A carbon fiber bar is used since it limits the effects of thermal expansion. The metrology bar markers on each end form a constant relative 6 DOF pose between the two ends. A shorter bar length should be used for smaller space measurements to maximize metrology bar measurements.

movement during dynamic

Most optical tracking systems have at least a 30 Hz data collection rate. Therefore, a minimum of 5 min of data needs to be collected. The workspace is uniformly divided by the artifact length. The artifact is moved using at least the minimum and maximum motion capture velocity specified for the system.

The static test procedure for measuring the performance of the optical tracking system is to divide the test space into a grid and place the artifact at intersections of the grid and at various orientations. The dynamic test procedure also divides the test space into a grid where the metrology bar is moved in a raster scan pattern forward-to-back and left-to-right throughout the space.

The metrology bar maintains a constant separation and orientation of the two marker clusters along all the paths and can be rigidly attached to and moved using a wheeled frame as illustrated in Figure 9 (b) that is pushed/pulled by a human, a mobile robot, or other mover to closely follow the path.

The metrology bar is moved at the maximum specified velocity of the optical tracking. Pose error measurement and reporting methods are also described in the ASTM WK49831 [ 8 ] working document.

III. CONCLUSION

The AGV standards development process has been limited for many years to considering only safety standards. Starting in late 2014, ASTM F45 Driverless Automatic Guided Industrial Vehicles performance standards are being developed to include navigation, docking, terminology and several other key areas for AGV’s, mobile robots, and mobile manipulators. As discussed in this paper, standard test methods for measuring vehicle performance are being developed so that manufacturers and users of these systems can easily replicate the measurements in their own facilities and at minimal cost and effort. More AGV and mobile robot systems, instead of just the one AGV used in these experiments, would ideally validate the generic test method proposed.

A comparison of GT measurement systems was also made to support the test method development. It was determined that for dynamic AGV measurement, an optical tracking system provided a suitable ground truth measurement. At the same time, a standard for these dynamic measurement Page 53 systems is also being developed. The standard will allow vehicle and robot performance standards developers to use the systems as ground truth with known measurement uncertainty. Optical tracking systems users and manufacturers can replicate the same test methods with similar tracking systems and use the results to compare their performance at dynamic tracking tasks.

ACKNOWLEDGMENT

The authors would like to thank the ASTM F45.02 subcommittee navigation task group and Omar Y. AboulEnein, Salisbury University student, for their recent input to navigation test method development and experimentation. Also, we thank Sebti Foufou, Qatar University, Doha, Qatar, for his guidance on the mobile manipulator docking performance measurement research.

References

[1] ANSI/ITSDF B56 . 5:2012, Safety Standard for Driverless, Automatic Guided Industrial Vehicles and Automated Functions of Manned Industrial Vehicles, www .itsdf.org, Nov 2012 .

[2] British Standard Safety of Industrial Trucks - Driverless Trucks and their Systems . Technical Report BS EN 1525 , 1998 .

[3] Bostelman , Roger, Will Shackleford, Geraldine Cheok, and Kamel Saidi . "Safe Control of Manufacturing Vehicle Research Towards Standard Test Methods." In Proc. International Material Handling Research Colloquium , pp. 25 - 28 . 2012 .

[4] Roger Bostelman , Elena Messina, “ Towards Development of an Automatic Guided Vehicle Intelligence Level Performance Standard”, Autonomous Industrial Vehicles: From the Laboratory to the Factory Floor , Chap. 1 , ASTM International , to be published 2015 .

[5] Roger Bostelman , Tsai Hong, and Geraldine Cheok, “ Navigation Performance Evaluation for Automatic Guided Vehicles” , IEEE International Conference on Technologies for Practical Robot Applications (TEPRA) , Boston, MA, April 2015 .

[6] ASTM International , Committee F45 on Driverless Automatic Guided Industrial Vehicles, www .astm.org/COMMITTEE/F45.htm, 2014 .

[7] Roger Bostelman , Will Shackleford, "Obstacle Detection and Avoidance from an Industrial Automatic Guided Vehicle," IROS 2014 .

[8] ASTM International , E57 .02 Standard Test Method for Evaluating the Performance of Rigid Body Tracking Systems that Measure Dynamic, Six Degrees of Freedom (6DOF), Pose , Work Item # WK49831 , June 2015 .

[9] ASTM International E2919 , Standard Test Method for Evaluating the Performance of Systems that Measure Static, Six Degrees of Freedom (6DOF) Pose , http://www.astm.org/Standards/ E2919-14.htm

[10] NDC 8 jAGV Control System , http://www.kollmorgen.com/enus/products/vehicle-controls/electrical-vehicle-controls/ndc8/, 2015 .

[11] Hvilshøj , Mads, and Simon Bøgh . "" Little Helper"-An Autonomous Industrial Mobile Manipulator Concept." International Journal of Advanced Robotic Systems 8 , no. 2 ( 2011 ).

[12] Roger Bostelman , Tsai Hong, Jeremy Marvel, “ Performance Measurement of Mobile Manipulators,” Proceedings SPIE DDS 2015 , Baltimore , MD , April 2015 .

[13] Burge , James H. , Peng Su , Chunyu Zhao , and Tom Zobrist . "Use of a commercial laser tracker for optical alignment." In Optical Engineering+ Applications , pp. 66760E - 66760E . International Society for Optics and Photonics , 2007 .

[14] Drury , Jill L., Brenden Keyes , and Holly Yanco . "LASSOing HRI: analyzing situation awareness in map-centric and video-centric interfaces." In Human-Robot Interaction (HRI), 2nd ACM /IEEE International Conference, pp. 279 - 286 . IEEE, 2007 .

[15] “Motion Capture Software Developers in the US: Market Research Report,” IBISWorld 2014 .