=Paper=
{{Paper
|id=Vol-1484/paper22
|storemode=property
|title=Intelligence Level Performance Standards Research for Autonomous Vehicles
|pdfUrl=https://ceur-ws.org/Vol-1484/paper22.pdf
|volume=Vol-1484
|dblpUrl=https://dblp.org/rec/conf/iros/BostelmanHM15
}}
==Intelligence Level Performance Standards Research for Autonomous Vehicles==
https://ceur-ws.org/Vol-1484/paper22.pdf
Intelligence Level Performance Standards Research for
Autonomous Vehicles
Roger B. Bostelman, Tsai H. Hong, and Elena Messina
Abstract— United States and European safety standards emergency response and military applications [4]. This
have evolved to protect workers near Automatic Guided reference also discusses research challenges, test and
Vehicles (AGV’s). However, performance standards for evaluations, and intelligent systems development programs
AGV’s and mobile robots have only recently begun that can support advancement of industrial AGVs towards
development. Lessons can be learned from research and attaining greater levels of intelligence. These other efforts also
standards efforts for mobile robots applied to emergency provide useful standards development criteria for AGV
response and military applications. Research challenges, performance test methods. Experiences and results in
tests and evaluations, and programs to develop higher advanced mobility and intelligence for robotics will be
intelligence levels for vehicles can also used to guide essential for AGV manufacturers and users to fully understand
industrial AGV developments towards more adaptable and capabilities and specific applications of their autonomous
intelligent systems. These other efforts also provide useful vehicle systems.
standards development criteria for AGV performance test
methods. Current standards areas being considered for Performance test methods for docking, navigation, (see
AGVs are for docking, navigation, obstacle avoidance, and Figure 1) [5], and terminology standard work items have been
the ground truth systems that measure performance. This initiated under the new ASTM Committee F45 on Driverless
paper provides a look to the future with standards Automatic Guided Industrial Vehicles performance standard
developments in both the performance of vehicles and the [6]. Standards for autonomous industrial vehicle obstacle
dynamic perception systems that measure intelligent vehicle avoidance and protection, based on past research [7],
performance. communication and integration, and environmental impacts
I. INTRODUCTION are also being considered.
Automatic Guided Vehicles (AGV’s) have typically been This paper will specifically discuss measurement of:
used for industrial material handling since the 1950’s. Since vehicle navigation (e.g., commanded vs. actual AGV path-
then, U.S. [1] and European [2] AGV safety standards have following deviation), vehicle docking (e.g., AGV stop point
evolved to protect nearby workers. These standards have positioning vs. known facility points), and obstacle detection
minimal test methods to describe how manufacturers and users and avoidance of standard test pieces (e.g., comparison of real-
are to perform AGV safety measurements, resulting in time AGV path-planning and new path following vs.
potential measurement differences across the industry. For commanded path) towards smart manufacturing applications,
example, American National Standards Institute/Industrial such as assembly and unstructured environment navigation.
Truck Safety Development Foundation (ANSI/ITSDF) Additionally, this paper will discuss a new ASTM Committee
B56.5:2012 provides new language to generically handle a on 3D Imaging Systems E57.02 [8] standard work item for six
situation when an object suddenly appears within the AGV degree-of-freedom (DOF) optical measurement of dynamic
stop region. The stop region is the area surrounding the AGV systems (see Figure 2), which advances the existing static 6
in which the non-contact safety sensor detects obstacles and DOF standard [9]. The new standard is expected to be a critical
stops the vehicle. The manufacturer must now prove that when component of performance measurement for current and
the AGV detects an object closer than its stopping distance, future robotic systems that rely on advanced perception
although collision with the object is perhaps imminent, the systems.
AGV demonstrates a reduction in kinetic energy. However, II. PERFORMANCE STANDARDS THRUSTS
there is no description of how manufacturers measure this
situation, resulting in different measurement results across AGV navigation, docking, and obstacle detection and
manufacturers. One test method was researched to handle this avoidance tests were conducted in support of future
situation and is described in [3]. performance standard test methods and are described in this
section. In some instances, typical industry practices were
Recently AGV and mobile robot performance standards evaluated as well as the improved AGV performance tests.
developments have begun to limit measurement method
differences. Initial developments began with a review of other
research and standards efforts for mobile robots as applied to
R. V. Bostelman is with the National Institute of Standards and T. H. Hong, is with the National Institute of Standards and Technology,
Technology, Gaithersburg, MD 20899, USA and with the IEM, Le2i, Gaithersburg, MD 20899, USA (phone: 301-975-3444; fax: 301-990-9688;
Université de Bourgogne, BP 47870, 21078 Dijon, France (phone: 301-975- e-mail: tsai.hong@nist.gov).
3426; fax: 301-990-9688; e-mail: roger.bostelman@nist.gov). E. Messina is with the National Institute of Standards and Technology,
Gaithersburg, MD 20899, USA (phone: 301-975-3510; fax: 301-990-9688;
e-mail: elena.messina@nist.gov).
FinE-R 2015 Page 48 IROS 2015, Hamburg - Germany
The path to success: Failures in Real Robots October 2, 2015
A. Vehicle Navigation
The most basic functions of mobile robots and AGV’s are
navigation to and docking with equipment in the workspace.
However, the description of how well the vehicle navigates
(i.e., commanded vs. actual AGV path-following deviation)
has certain ambiguities. For example, navigation implies that
the vehicle measures its current position, plans a route to
another location, and moves from the current location to
planned location upon command. Most vehicle
manufacturers don’t provide specifications for how uncertain
the navigation performance is (i.e., the error bounds on
position or velocity), other than perhaps radius of vehicle
turns, maximum velocity, and maximum acceleration. The
vehicle velocity sets limits on the allowable turn radius for
particular vehicles. Some controllers [10], if not all, will not
allow high velocities on relatively small radii to prevent Figure 1. Example reconfigurable apparatus for navigation tests for various
unsafe vehicle conditions. These limitations are not typically AGV sizes.
specified by AGV manufactures, causing AGV users To address AGV navigation uncertainty, with an eye
difficulty in planning how many vehicles they may require for towards a potential test method for all automatic industrial
moving their products within the facility to maintain a desired vehicles, tests were executed, both with an AGV prior to and
throughput. after being calibrated. The uncalibrated AGV test is similar to
Industrial vehicles may eventually become uncalibrated typical industry methods since not all AGVs can be frequently
through regular use. An uncalibrated vehicle does not follow calibrated. An uncalibrated AGV was moved along a straight
a commanded path or stop/dock at a commanded point with line path between two commanded points in an open area and
minimal relative uncertainty (standard deviation of measured spaced approximately 5 m apart [5]. Figure 2 shows the results
vs. ground truth) as does a calibrated vehicle. To correct this, amplified in the X direction 100 times to exaggerate vehicle
vehicle manufacturers have calibration procedures for their performance. In the figure, the blue line is the commanded
vehicles, although these procedures can be tedious, time- path between points 1 and 2. The green dots to the right and
consuming, and may not be appropriate for all vehicles. For left of the line are uncalibrated AGV controller-traced position
example, calibration of Ackerman steered vs. ‘crab’ steered data moving forward and reverse, respectively, between the
(sometimes called quad) vehicles have different calibration points. The red dots are ground truth of the navigating AGV
procedures. It is not always clear what will happen when a between points using an optical tracking system. This
vehicle is uncalibrated nor when the vehicle becomes experiment demonstrated one AGV navigation performance
uncalibrated. The effects of calibration on vehicle control and measurement method using a precision (0.2 mm standard
uncertainty are typically not specified either. There is also deviation) six degree-of-freedom (DOF), optical measurement
typically no specification describing how far from the system as a ground truth comparison to the onboard vehicle
commanded path a vehicle navigates. This may be important tracking system. Path deviation was approximately 20 cm
to users who have tight tolerance AGV paths (e.g., paths maximum. The AGV was then calibrated using the
between infrastructure) that must be followed. A test can be manufacturer’s method.
developed to uncover the effects of uncalibrated vs. calibrated
vehicle navigation performance when commanded to move
along a path, as shown as a dashed line in the example in
Figure 1. Should objects be near the vehicle path, such as walls
or obstacles, depicted in Figure 1 as bordering lines along the
path, the vehicle may stop, slow, or worse, collide with the
boundary object. A user would then be required to provide Pt 2
additional, perhaps unnecessary space for one manufacturers’
vehicle and not for another. How the vehicle handles (slow,
stop, etc.) the event is also ambiguous. For example, some, but
not all vehicles are equipped with obstacle detection based on
non-contacting sensors that provide detection beyond the
physical vehicle footprint.
Pt 1
Figure 2. Ground Truth (red) and AGV (green) data of the straight line path
tests. Scales for X and Y axes are in meters where the X axis shows only -
0.11 to -0.02 range to clearly show the AGV performance as compared to
Ground Truth measurement. The blue line represents the commanded path
from pt 1 to pt 2 and back.
FinE-R 2015 Page 49 IROS 2015, Hamburg - Germany
The path to success: Failures in Real Robots October 2, 2015
Another test setup was tried, with an eye towards a A series of eight trials were completed with nearly all trials
relatively less expensive test method that will allow all AGV including three or more runs each to demonstrate the
systems to be measured, ideally, with an independent navigation test method concept. Ten or more runs are ideal for
measurement method that doesn’t use AGV controller statistical analysis. The optical measurement system
tracking, yet captures the full AGV configuration (i.e., mentioned earlier was used as an experimental ground truth
including safety sensing). The AGV was commanded to drive (GT) to measure the barrier and vehicle position during
back and forth between temporary barriers, along a straight experiments to further understand the test method and vehicle
line defined by commanded points spaced approximately 10 m performance. The barriers and AGV were marked with
apart. The goal of the experiment was to measure the AGV spherical reflectors (visible in Figure 4 (a, b, and c) detectable
deviation from the commanded path. A critical AGV from the GT system. Figure 5 presents GT data plotted for
navigation performance area is also deviation from the navigation tests showing ground truth data of: (a) test 8 vehicle
commanded path after turns so a 90° turn was added to the end path and emergency stopped vehicle (red circle) when a wall
of the straight path beyond the barriers to measure the vehicle was detected, (b) test 1 path, and (c) test 1 path data from (b)
navigation uncertainty when moving from/to a straight path zoomed in to show data points of three runs.
to/from a turn. Figure 3 shows the test setup and Figure 4
shows (a) a B56.5 test piece being used to define the safety
laser stop field edges, (b) the barriers and lines to which
barriers are moved between trials, and (c) the AGV
emergency-stopped upon detection of the barriers. The safety
laser, stop field edges were marked on the floor, as a ground
truth, zero-tolerance spacing that the vehicle can navigate,
when the vehicle was at position 1 and again at position 3,
shown in Figure 3, for both left and right vehicle sides. The
barrier position lines were measured from the edge line using
a ruler and marked at 2 cm increments from the edge up to 10 blue barrier-
cm away from the edge line. Smaller spacing between lines position lines
(e.g., 1 cm) could also be used for finer uncertainty
measurement. For each test trial, the barriers were moved a b c
towards the AGV to the next line beginning at 10 cm for trial Figure 4. (a) B56.5 test piece (black cylinder) used to define safety laser edge
1, 8 cm for trial 2, and so forth until the navigating vehicle (note red emergency stop light (within the red circles) is on), (b) barrier
detected a barrier, and emergency-stopped the AGV, thus (black) painted wood panel, blue lines spaced at 2 cm, and spherical reflector
completing the test run. from ground truth system, (c) AGV emergency stopped, as noted by the red
light, upon detection of barriers during a test.
Experimental results from the barriers demonstrated a path
uncertainty of between 6 cm and 8 cm maximum when the
vehicle detected the boundaries at nearly the center of the
straight line path and when moving at either 0.25 m/s or 0.50
m/s. The navigation test method using barriers is simple and
cost-effective for manufacturers and users to employ, as
compared to the higher accuracy, but more expensive ground
truth visual tracking system used for test method development.
A simple straight line with one turn was tested. However,
more complex test configurations, such as shown in Figure 1,
could be set up using B56.5 test pieces instead of larger,
physical barriers as were used in this research.
Figure 3. AGV navigation test setup.
Figure 5. Example graphical results of navigation tests showing ground truth
data of: (a) test 8 vehicle path and emergency stopped vehicle (red circle)
FinE-R 2015 Page 50 IROS 2015, Hamburg - Germany
The path to success: Failures in Real Robots October 2, 2015
when a wall was detected, (b) test 1 path, and (c) test 1 path data from (b) point, whereas the visual tracking system can track multiple
zoomed in to show (red, green and blue) data points from three runs. point markers and can computer orientation from them. Both
A working document that addresses quantifying vehicle GT systems can measure relatively high-precision
navigation uncertainty is being developed as an initial step displacement between two points, as compared to an AGV
towards a performance standard for ASTM F45.02 docking.
subcommittee on Docking and Navigation. Based on An experiment using an uncalibrated AGV that was
consensus of the task group developing this standard, as was programmed to stop at various points yielded an uncertainty
tested at NIST, the simple path-bounding test method using range of approximately 1 mm to 50 mm. Figure 7 (a) shows
temporary reconfigurable barriers made from readily- the vehicle paths and Figure 7 (b) shows average errors for five
available, off-the-shelf materials is being proposed. runs at stop or dock points. The vehicle position was measured
B. Vehicle Docking using a laser tracking GT system which provided high-
precision measurement of AGV stop points. [13] However, in
Vehicle docking is another common application of mobile
several experiments, laser tracker positioning was critical as
robots and AGVs. Unit load (tray, pallet, or cabinet carrying),
the laser beam was continuously interrupted by onboard AGV
tugger (cart pulling), and fork/clamp (pallet or box
hardware. This prompted a switch to using an optical tracking
load/unloading) are typical industrial style vehicles that
system for GT measurements.
require different docking uncertainties. For example, a unit
load vehicle that places/retrieves platters during wafer A 6 DoF optical tracking GT system was used instead to
manufacturing would no doubt require less uncertainty than a measure AGV docking. Docking was measured again after the
fork style vehicle that places/retrieves pallets. As robotics AGV was calibrated using the manufacturer’s procedures. The
advances, current and potential users are requesting mobile AGV approached similar dock locations and after AGV
manipulators to perform tasks such as unloading trucks. calibration, provided consistent 5 mm uncertainty. Standards
Eventually, it is expected that mobile manipulators will be development for optical tracking systems is also underway and
used for smart manufacturing assembly applications [11, 12]. is discussed in section 2 D, 6 DOF Optical Measurement of
Dynamic Systems.
Similar to navigation, there are no performance
measurement test methods that define how manufacturers and
users characterize their vehicle’s docking capabilities. Figure
6 (a) shows an example method for docking for any style
vehicle. A vehicle approaches and makes contact with ‘a’
and/or ‘b’ docking points dependent upon the vehicle type.
Relative displacement from each of the points would be
measured to determine vehicle docking uncertainty. A fork-
type AGV is shown docked with a test apparatus in Figure 6
(b). The fork tips are marked with yellow points.
(a)
(a) (b)
Figure 6. (a) Example docking test method using various AGVs (e.g., 1 and 2
for AGV unit load tray table docking, 3 for fork and tugger AGV docking).
“a” and “b” are fixed points in space (e.g., contact or non-contact sensor
Docking points
locations in space). Approach vectors and sensor point spacing and locations
are variable. (b) Fork-type AGV docking with a docking apparatus. (b)
Two experiments were simultaneously performed: AGV Figure7. (a) Commanded paths and stop points and (b) stop point errors of a
docking relative to known facility locations and GT system use single AGV point for each location in (a) averaged over 5 runs.
for measuring AGV docking. Two different GT measurement Additional AGV equipment docking experiments were
systems were used to measure AGV performance: a laser also performed using a mobile manipulator and a
tracking GT with an uncertainty of approximately 10 µm [13] reconfigurable mobile manipulator artifact (RMMA)
and an optical tracking system with uncertainty of 0.2 mm in developed at NIST (see Figure 8). [14] The mobile
position uncertainty and 0.13° in angle uncertainty as manipulator, with uncalibrated AGV, repeatedly moved next
measured at NIST. The laser tracker tracks position of a single to the artifact from a starting point. Although uncalibrated, the
FinE-R 2015 Page 51 IROS 2015, Hamburg - Germany
The path to success: Failures in Real Robots October 2, 2015
AGV provided relatively low repeatability uncertainty (e.g.,
+/-5 mm) although more than 10 mm from the commanded
docking points. This manipulator could reach the commanded
points on the RMMA even with 10 mm uncertainty in AGV
position. The mobile manipulator corrected for the position
uncertainty after being taught the actual RMMA locations. At
the RMMA, the manipulator, wielding a laser retroreflector,
was commanded to move in a spiral pattern to detect 6 mm Figure 9. Graphical output of path planner, starting footprint of the AGV is in
white, the goal position is a dark grey rectangle. Yellow rectangles show the
diameter reflectors. The reflectors provide non-contact area swept out as the AGV would travel, blue curve shows the resulting
alignment detection of the tool point position and orientation. spline, and orange circles represent obstacles.
The experiment provided results demonstrating that this
relatively inexpensive ground truth measurement method was The navigation performance measurement experiment
sufficient for measuring docking accuracy. As the reflector discussed previously in section II A. Vehicle Navigation can
based measurement system is inexpensive compared to the be similarly applied for obstacle detection and avoidance. In
optical tracking-based GT, it may prove ideal for use as a fact, the ASTM F45.02 subcommittee navigation and docking
precision vehicle/mobile manipulator docking test method that task groups have discussed the potentially overlapping nature
both manufacturers and users can replicate. of the two vehicle capabilities. The ASTM F45.03 Obstacle
Detection and Protection subcommittee is currently in the
process of considering standards in this area. Questions have
been raised regarding standards development as follows:
Manipulator
1. How well does the AGV react to situations? For
example:
RMMA Obstacles appearing in the path
Potential obstacles headed towards the path
Unstructured (i.e., changing obstacle locations)
AGV areas not on the original planned path or that
rapidly change
2. How far off the commanded navigation path can an
AGV be, and at what speeds, before it violates the path
and causes a stop? For example, due to environmental
factors such as:
Offset-pitched/rolled AGV can’t see guidance
markers, such as reflectors, magnets, wire, etc.
Figure 8. Docking performance measurement of a mobile manipulator with a
Guidance or boundary-marking tape is worn or
reconfigurable mobile manipulator artifact (RMMA). broken
Terrain causes “bouncing” or moving laser or other
C. Obstacle Detection and Avoidance
navigation sensors
Obstacle detection and avoidance (ODA) research is well 3. How well does the vehicle react when a human is
documented in the literature for mobile robots. However, detected and how should the human be represented? For
there are few citations for AGVs perhaps due to the relatively example:
closed nature of commercially available AGV controllers and
By test pieces, mannequins, humans
because ODA is not often implemented on AGVs deployed in
large manufacturing facilities. In [5], it was discussed that for With what coverings? (i.e., what clothes should be
large facilities, ODA could occur in ‘buffer zones’ (i.e., zones worn?)
where AGVs would be allowed to pass other vehicles). For 4. How to interact with manual equipment (e.g., forklifts,
small and medium manufacturing facilities, however, ODA machines)
may be necessary due to more limited floor space and less- 5. How to standardize communication of vehicle
controlled environments. NIST has developed an algorithm, intelligence for obstacle detection and avoidance? For
detailed in [5], and measured the performance of an AGV with example:
added ODA capability. The algorithm is also suitable for Contextual autonomy levels [4]
navigating an unstructured environment although it is Situation awareness (e.g. LASSO) [14]:
currently limited by the use of facility-mounted (sensors not
mounted on the AGV) obstacle detection with obstacle Experiments to support ODA performance test method
avoidance adapted to an AGV with a controller with limited development will be performed based on forthcoming
ability to integrate external algorithms. Figure 9 shows a guidance from the ASTM F45 subcommittee. However, a
snapshot of the ODA algorithm planning a path through prototype safety test method that has been developed to
multiple obstacles. evaluate a vehicle’s response to obstacles in its path and within
its stop zone, as noted in the Introduction, can be considered a
first step towards full ODA standard test methods. ASTM F45
is meant to dovetail with safety standards such as
ANSI/ITSDF B56.5. Therefore, providing an initial test
FinE-R 2015 Page 52 IROS 2015, Hamburg - Germany
The path to success: Failures in Real Robots October 2, 2015
method for detection of obstacles is ideal as a starting point for maximize metrology bar movement during dynamic
F45.03. The ‘Grid-Video’ detection method [3] provides a measurements.
simple-to-implement test method that measures positional
accuracy of the dynamic test piece relative to the vehicle
position when the obstacle enters the vehicle path.
D. 6 DOF Optical Measurement of Dynamic Systems
ASTM’s draft Standard for the Performance of Optical
Tracking Systems that Measure Static and Dynamic Six
Degrees of Freedom (6DOF) Pose (see Figure 10) is the next
step beyond the static case covered by ASTM E2919-14 [8].
Optical tracking is being used for robot and autonomous
vehicle GT measurement, as discussed in this paper. Optical
tracking measurement systems [15] are used in a wide range
of fields, including video gaming, filming, neuroscience,
biomechanics, flight/medical/industrial training, simulation, Figure 9. (a) Proposed metrology bar, (b) Example frame used to move the
metrology bar.
and robotics. ASTM WK49831 is a working document that is
considering both static and dynamic measurements of systems Most optical tracking systems have at least a 30 Hz data
under test. The scope of the draft standard test method is to collection rate. Therefore, a minimum of 5 min of data needs
provide metrics and procedures to determine the performance to be collected. The workspace is uniformly divided by the
of a rigid object tracking system in measuring the dynamic artifact length. The artifact is moved using at least the
pose (position and orientation) of an object. Optical minimum and maximum motion capture velocity specified for
measurement systems may use the test method to establish the the system.
performance for their 6 DOF rigid body tracking pose
The static test procedure for measuring the performance
measurement systems. The test method will also provide a
of the optical tracking system is to divide the test space into a
uniform way to report the statistical errors and the pose
grid and place the artifact at intersections of the grid and at
measurement capability of the system, making it possible to
various orientations. The dynamic test procedure also divides
compare the performance of different systems. So all the
the test space into a grid where the metrology bar is moved in
measurements can be traced to the standard.
a raster scan pattern forward-to-back and left-to-right
Ground truth throughout the space.
cameras
The metrology bar maintains a constant separation and
orientation of the two marker clusters along all the paths and
can be rigidly attached to and moved using a wheeled frame
as illustrated in Figure 9 (b) that is pushed/pulled by a human,
a mobile robot, or other mover to closely follow the path.
The metrology bar is moved at the maximum specified
AGV velocity of the optical tracking. Pose error measurement and
reporting methods are also described in the ASTM WK49831
[8] working document.
III. CONCLUSION
The AGV standards development process has been limited
for many years to considering only safety standards. Starting
in late 2014, ASTM F45 Driverless Automatic Guided
Industrial Vehicles performance standards are being
Figure 10. (top) autonomous vehicle test lab and (bottom) screenshot of the developed to include navigation, docking, terminology and
perception ground truth system space showing cameras and vehicle rigid several other key areas for AGV’s, mobile robots, and mobile
body. manipulators. As discussed in this paper, standard test
In the initial test procedure, measurements with methods for measuring vehicle performance are being
uncertainties were computed using an artifact – namely a developed so that manufacturers and users of these systems
metrology bar as shown in Figure 9 (a). Current optical can easily replicate the measurements in their own facilities
tracking systems utilize a three-marker metrology bar with all and at minimal cost and effort. More AGV and mobile robot
markers in a line which does not provide 6 DOF system systems, instead of just the one AGV used in these
performance measurement. A metrology bar made of carbon experiments, would ideally validate the generic test method
fiber with length 620 mm and with five reflective markers proposed.
attached on each end was used as the 6 DOF artifact. A carbon A comparison of GT measurement systems was also made
fiber bar is used since it limits the effects of thermal to support the test method development. It was determined
expansion. The metrology bar markers on each end form a that for dynamic AGV measurement, an optical tracking
constant relative 6 DOF pose between the two ends. A shorter system provided a suitable ground truth measurement. At the
bar length should be used for smaller space measurements to same time, a standard for these dynamic measurement
FinE-R 2015 Page 53 IROS 2015, Hamburg - Germany
The path to success: Failures in Real Robots October 2, 2015
systems is also being developed. The standard will allow
vehicle and robot performance standards developers to use the
systems as ground truth with known measurement
uncertainty. Optical tracking systems users and manufacturers
can replicate the same test methods with similar tracking
systems and use the results to compare their performance at
dynamic tracking tasks.
ACKNOWLEDGMENT
The authors would like to thank the ASTM F45.02
subcommittee navigation task group and Omar Y. Aboul-
Enein, Salisbury University student, for their recent input to
navigation test method development and experimentation.
Also, we thank Sebti Foufou, Qatar University, Doha, Qatar,
for his guidance on the mobile manipulator docking
performance measurement research.
REFERENCES
[1] ANSI/ITSDF B56.5:2012, Safety Standard for Driverless, Automatic
Guided Industrial Vehicles and Automated Functions of Manned
Industrial Vehicles, www.itsdf.org, Nov 2012.
[2] British Standard Safety of Industrial Trucks - Driverless Trucks and
their Systems. Technical Report BS EN 1525, 1998.
[3] Bostelman, Roger, Will Shackleford, Geraldine Cheok, and Kamel
Saidi. "Safe Control of Manufacturing Vehicle Research Towards
Standard Test Methods." In Proc. International Material Handling
Research Colloquium, pp. 25-28. 2012.
[4] Roger Bostelman, Elena Messina, “Towards Development of an
Automatic Guided Vehicle Intelligence Level Performance Standard”,
Autonomous Industrial Vehicles: From the Laboratory to the Factory
Floor, Chap. 1, ASTM International, to be published 2015.
[5] Roger Bostelman, Tsai Hong, and Geraldine Cheok, “Navigation
Performance Evaluation for Automatic Guided Vehicles”, IEEE
International Conference on Technologies for Practical Robot
Applications (TEPRA), Boston, MA, April 2015.
[6] ASTM International, Committee F45 on Driverless Automatic Guided
Industrial Vehicles, www.astm.org/COMMITTEE/F45.htm, 2014.
[7] Roger Bostelman, Will Shackleford, "Obstacle Detection and
Avoidance from an Industrial Automatic Guided Vehicle," IROS 2014.
[8] ASTM International, E57.02 Standard Test Method for Evaluating the
Performance of Rigid Body Tracking Systems that Measure Dynamic,
Six Degrees of Freedom (6DOF), Pose, Work Item #WK49831, June
2015.
[9] ASTM International E2919, Standard Test Method for Evaluating the
Performance of Systems that Measure Static, Six Degrees of Freedom
(6DOF) Pose, http://www.astm.org/Standards/ E2919-14.htm
[10] NDC 8 jAGV Control System, http://www.kollmorgen.com/en-
us/products/vehicle-controls/electrical-vehicle-controls/ndc8/, 2015.
[11] Hvilshøj, Mads, and Simon Bøgh. "" Little Helper"-An Autonomous
Industrial Mobile Manipulator Concept." International Journal of
Advanced Robotic Systems 8, no. 2 (2011).
[12] Roger Bostelman, Tsai Hong, Jeremy Marvel, “Performance
Measurement of Mobile Manipulators,” Proceedings SPIE DDS 2015,
Baltimore, MD, April 2015.
[13] Burge, James H., Peng Su, Chunyu Zhao, and Tom Zobrist. "Use of a
commercial laser tracker for optical alignment." In Optical
Engineering+ Applications, pp. 66760E-66760E. International Society
for Optics and Photonics, 2007.
[14] Drury, Jill L., Brenden Keyes, and Holly Yanco. "LASSOing HRI:
analyzing situation awareness in map-centric and video-centric
interfaces." In Human-Robot Interaction (HRI), 2nd ACM/IEEE
International Conference, pp. 279-286. IEEE, 2007.
[15] “Motion Capture Software Developers in the US: Market Research
Report,” IBISWorld 2014.
FinE-R 2015 Page 54 IROS 2015, Hamburg - Germany
The path to success: Failures in Real Robots October 2, 2015