The reliable DNN-based perception of pedestrians represents a crucial step towards automated driving systems. Currently applied metrics for a subset-based evaluation prohibit an application-oriented performance evaluation of DNNs for pedestrian detection. We argue that the current limitation in evaluation can be mitigated by the use of image segmentation. In this work, we leverage the instance and semantic segmentation of Cityscapes to describe a rule-based categorization of potential detection errors for CityPersons. Based on our systematic categorization, the filtered log-average miss rate as a new performance metric for pedestrian detection is introduced. Additionally, we derive and analyze a meaningful upper bound for the confidence threshold. We train and evaluate four backbones as part of a generic pedestrian detector and achieve state-of-the-art performance on CityPersons by using a rather simple architecture. Our results and comprehensible analysis show benefits of the newly proposed performance metrics. Code for evaluation is available at https://github.com/BeFranke/ErrorCategories.
Pedestrian detection is a crucial perception task for automated driving systems (ADS). Due to high complexity of the ADS environment, supervised machine learning models such as deep neural networks (DNNs) outperform traditional computer vision models and meet the high performance standards. Hence, traditional methods such as HOG [1] have been replaced by DNNs, which can be designed single-staged and anchor-free [2, 3] or two-staged and anchor-based [4].
Avoiding false negatives is the main objective for pedestrian detection in an ADS. A critical scene as shown in Figure 1 outlines the key task: A group of pedestrians cross the street right in front of the automated vehicle (AV). Intuitively, the evaluation should focus on pedestrians in the immediate vicinity of an AV, rather than distant pedestrians standing on the sidewalk in the middle of a crowd. The goal is to build relevant subsets of an evaluation dataset that contains these highly safety-critical pedestrians. This enables a more meaningful performance evaluation of DNNs. Motivated by the Caltech evaluation protocol [5], occlusion-related The IJCAI-ECAI-22 Workshop on Artificial Intelligence Safety †These authors contributed equally.
Despite eforts to address highly occluded and therefore very dificult pedestrian detection cases, we argue that a realistic performance evaluation of a DNN for pedestrian detection should primarily address pedestrians in the near field of an AV. In this sense, we think that a missed pedestrian in a distant crowd is less significant than a missed pedestrian standing directly in front of the AV.
Although a high recall is the primary objective for a DNN in a safety-critical application, the precision strongly influences the ADS operation in a complex environment. The current subset-based evaluation neglects the impact of diferent forms of false positives. We argue that multiple detections of the same pedestrian are less problematic for an ADS than false positives randomly scattered in the scene without reference to pedestrianlike features. Thus, a clear distinction between false positives must be found.
In our work, we introduce a systematically derived categorization of errors that can leverage a safety argumentation for the DNN-based perception of pedestrians.
Our contribution can be summarized as follows: 1. We propose a rule-based categorization that describes potential errors of a DNN for pedestrian detection. Feature Extraction Pre-trained image classification 2. We define novel performance metrics focusing on networks form the backbone of the feature extraction. safety-critical pedestrians that enable application- To utilize backbones for pedestrian detection, additional oriented DNN evaluation. layers (ALs) must be implemented. Based on computed 3. We report results and analyze 44 diferent DNNs features for various scales by the backbone, the feature exfor pedestrian detection, divided into 11 training traction outputs a representation for a given input image. runs for four backbones. In our work, we use the following feature extractions:
Pedestrian detection is usually done by locating a 2d bounding box and assigning the correct class. Most commonly, DNNs for pedestrian detection are evaluated with the log-average miss rate (LAMR) [10] on the reasonable subset of the CityPersons [11] validation dataset. We refer to this performance metric as LAMRr. Since pedestrian detection is highly safety-critical and relevant to an ADS, the LAMRr aggregates the miss rate (MR) and false positives per image (FPPI).
Table 1 gives an overview of state-of-the-art DNNs for pedestrian detection that are evaluated on diferent subsets of CityPersons. The definitions of the subsets are based on the height interval ℎ = [50, 1024] and a varying visibility range = | R|RGG | | of a pedestrian: reasonable ( = [0.65, 1] ), bare ( = [0.90, 1] ), partial ( = [0.65, 0.90] ) and heavy ( = [0, 0.65] ).
In this work, we provide a comprehensive analysis of diferent backbones that are commonly used for DNNs for pedestrian detection. To achieve comparable results, • CSP-ResNet-50: CSP [2] creates high-level semantic features based on ResNet-50 [18] and deconvolutions. • FPN-ResNet-50: Feature pyramid network (FPN) [3] that adds a pyramidal decoder to ResNet50 to combine features from diferent scales. • MDLA-UP-34: Modified DLA (MDLA) [ 19] augmentes DLA-34 [20] with deformable convolutions from lower layers to the output. • BGC-HRNet-w32: BGC [17] adds deconvolutions to a HRNet-w32 [21] concatenating the outputs to form the final representation.
Perception Heads In total, we have three perception heads taking extracted features as inputs and outputting a center, scale (height w/o width) and ofset map. Similar to APD [3], we apply 3x3 convolutions for each perception head.
Training We train and evaluate diferent GPD instances with varying pre-trained backbones on the CityPersons dataset. In the following, an instance of GPD is simply referred to as a pedestrian detector (PD).
All PDs are trained with the Adam optimizer [22] without weight decay and a reduced image size of 640x1028 pixels.
A linear warm up strategy is employed that increases the
G Reasonable learning rate from 5 ⋅ 10−8 to the final learning rate of 10−4 over 2k iterations. We train for a maximum of 50k iterations on 2 GPUs with a batch size of 8. The final PD is given by the best checkpoint with the lowest LAMR score on the reasonable subset of the CityPersons validation dataset. ResNet-501, DLA-342 and HRNet-w323 are used as pre-trained backbones on ImageNet. Furthermore, we apply the center, scale and ofset loss terms according to CSP [2]. Common data augmentation techniques like modifying brightness, contrast or saturation are applied.
Bounding Boxes We define a bounding box R as set of all pixels with (, ) corner coordinates that fall into the Inference For post-processing, we apply a confidence bounding box: R = {(, ) ∣ 1 ≤ < 2 ∧ 1 ≤ < 2}. threshold of 0.01 and use NMS with a threshold of 0.5. Therefore, the width of the bounding box is defined as The inference of PDs is conducted with the original image ( R) = 2 − 1 and similarly the height ℎ(R) = 2 − 1. size of 1024x2048 pixels. Ground truth and detection A ground truth bounding box is denoted as RG ∈ in bounding boxes are clipped to the image size. contrast to a detection bounding box ̃ ∈ ̃ which is associated with a confidence score ( RD). Because of 4. Methodology highly overlapping detections, post-processing methods such as non-maximum suppression (NMS ∶ →̃ ) In the following, we introduce diferent categories for are applied to reduce the number of detections to RD ∈ ground truth bounding boxes RG of the CityPersons val- . Based on a predefined confidence threshold , less idation dataset. Matching ground truth with detection confident detections are ignored: D(c) = {RD ∣ RD ∈ bounding boxes RD based on our systematic categoriza- ∧ ( RD) > }. tion identifies errors for false negatives and false pos- Generally, a pixel-precise match between bounding itives. Reducing false negatives is the primary safety- boxes can not be expected. Therefore the intersection related objective during PD training. Intuitively, we over union (IoU) is used to measure the localization qualexpect false negatives to be positively correlated with ity of RD for RG. The set of true positives is defined pedestrian occlusion by other pedestrians and other envi- as: ronmental objects. That’s why categories regarding false negatives build upon the description of diferent forms of occlusions. As shown in Figure 2 and Figure 3, our TPG(c) ={RG ∣ RG ∈ ∧ ∃ RD ∈ D(c) ∶ proposed error categorization recovers ignored pedestri- [IoU(RG, RD) > 0.5 ∧ ∄R̃G ∈ ∶ (1) ans for the reasonable and bare subset of the CityPersons validation dataset. Finally, we categorize false positives IoU(R̃G, RD) >IoU(RG, RD)]} to identify the most disruptive ones for an ADS.
A ground truth bounding box RG that can not be matched is a false negative FNG(c) = ⧵ TPG(c). A detection bounding box RD that can not be matched or can only be matched to an already matched RG is assigned to the set of false positives: define O as 20 selected classes of the semantic segmentation of Cityscapes [23] that can potentially cause occlusion. Truncated bounding boxes belong to this category, as the area that extends beyond the image is understood as environmental occlusion. We define the visibility with respect to the environment as (RG) = | RG ∩ {(, ) ∣ [, ] ∈
O}| | RG | (4)
For our experiments, we empirically set = 0.7.
We define the intermediate set of environmentally occluded ground truth bounding boxes as ℰ̃ = {RG ∣ RG ∈ FPD(c) ={RD ∣ RD ∈ D(c) ∧ ∄RG ∈ ∶
IoU(RG, RD) > 0.5 ∨ ∃RG ∈ ∶ [IoU(RG, RD) > 0.5 ∧ ∃R̃D ∈ ∶
IoU(RG, R̃D) >IoU(RG, RD)]} Image Segmentation In this work, we employ ground truth for semantic segmentation and instance segmentation ℑ to refine the subset-based evaluation of DNNs for pedestrian detection. [, ] = person means that the pixel at position (, ) belongs to a pedestrian. ∧̃ (RG) > }. ℑ[, ] = means that the pixel at position (, ) has the instance ID .
We define five error categories that separate ground truth bounding boxes of occluded pedestrians as well as highly safety-relevant pedestrians standing in the foreground or background. Examples of our categorization are shown in Figure 4. We propose a two-stage process to detect occlusion. First, potentially occluded pedestrians are identiifed based on the segmentation-based visibility , where represents the instance ID belonging to the pedestrian: ( RG, ) = | RG ∩ {(, ) ∣ ℑ[, ] = }| | RG |
Crowd Occlusion Crowd occlusion (also intra-class occlusion [7]) occurs when a pedestrian is occluded by other pedestrians. We define the intra-class visibility that describes the relation of the instance area of a pedestrian to the semantic area occupied by the person class:
The set of occlusion candidates ̃ builds upon the threshold : =̃ { RG ∣ RG ∈ ∧ (RG) < }. For our experiments, we empirically set = 0.6.
Ambiguous Occlusion Ambiguous occlusion occurs when pedestrians are simultaneously occluded by the environment and other pedestrians. We introduce the ambiguity factor ∈ (0, 1)to relax the thresholds for Environmental Occlusion Environmental occlusion crowd and environment occlusion and define = {RG ∣ occurs when a pedestrian is partially hidden behind ob- RG ∈ ℰ̃ ∧ (RG) > ⋅ } and = {RG ∣ RG ∈ jects in the scene, e.g. trafic signs, vegetation or cars. We ∧̃ (RG) > ⋅ }. (RG, ) =
| RG ∩ {(, ) ∣ ℑ[, ] = }| | RG ∩ {(, ) ∣ [, ] = person}| (5) (3)
We introduce the threshold and define the intermediate set of crowd occluded ground truth bounding boxes as =̃ { RG ∣ RG ∈ ̃∧ (RG, ) > }. For our experiments, we empirically set = 0.5.
ID(c * ) TPD(c * ) S(c * ) L(c * ) H(c * ) detections ID(c). TPD(c)(solid), ghost detections H(c)(dash dotted), localization errors L(c)(dashed), scale errors S(c)(dotted) and ignored Ground truth bounding boxes with ambiguous occlu- specified as ℬ = ⧵ ℱ
and potential background errors are defined as:
B(c) =FNG(c) ∩ ℬ.
Due to highly crowd-occluded pedestrians that introduce doubtful false negatives into the evaluation, we relax the matching strategy in Equation 1: For all RG ∈ ℱ ∪ ℬ, we see RG as a true positive if there exists a detection with IoU > 0.5 irrespective of another RG ∈ that could be matched with a higher IoU. sion are defined as =
∪ . Based on that, the set of environmentally occluded ground truth bounding boxes is reduced to ℰ = ℰ̃ ⧵ and the set of crowd occluded ground truth bounding boxes is = experiments, we empirically set = 0.75.
⧵̃
. For our the foreground ℱ = {RG ∣ RG ∈ ∧ Foreground and Background
After defining occluded ground truth bounding boxes as = ℰ ∪ ∪ the clearly visible bounding boxes are given by = ⧵ By applying a height threshold , we can further divide into foreground ℱ or background ℬ. First, we define
, height(RG) ≥ }.
Then, F(c) =FNG(c) ∩ ℱdefines errors in the foreground.
In order to define a reasonable , the braking distance of an automated emergency braking AEB is defined as AEB = + + ⌈ ⌉ + ⌈ ⋅
⌉. an automated emergency braking AEB with 30 ℎ .
Description Pedestrian height Processing time Friction coeficient Velocity Gravitational constant Added distance
Distance from rear axis to front
Applying the parameters shown in Table 2, the separating distance is AEB = 22 . Based on camera calibration parameters of Cityscapes, the corresponding pixel height is approximately 190 pixels. Finally, the background is
Parameter ℎ 2 2⋅⋅ Value 1.7 0.4 0.3 8.33 9.81 2
2 4 . 4.2. Error Categories for False Positives
A detection bounding box RD ∈ FPD(c)is a false positive.
We argue that false positives that coincide with pedestrian crowds do not disrupt the operation of an ADS as much as unrelated and random false positives. Hence, we propose three error categories with respect to false positives in order to identify the most disruptive. For examples see Figure 5. of RD is aligned with RG, is defined as: Scale Errors
fail only with respect to the scale of the bounding box. Let cX(R),cY(R) give the x- and y-center coordinates of any bounding box R and the maximum permitted center ofset. The predicate that states whether the center D(RG, RD) ⟺ |cX(RG) −cX(RD)| ≤ ( RG) ∧ | cY(RG) −cY(RD)| ≤ ℎ(RG) (6) ∶
D(RG, RD)}.
For our experiments, we empirically set = 0.2. Scale errors are defined as S(c) = {RD ∣ RD ∈ FPD(c) ∧ ∃RG ∈ Localization Errors
in close proximity to a RG, but the detection can not be matched and is not a scale error. Localization errors
In the following, we derive metrics to measure the performance of a PD over the proposed error categories. Table 4 shows the number of ground truth bounding boxes for each category. The filtered miss rate MRP(c)accounts for ground truth bounding boxes with P ∈ {ℱ, ℬ, ℰ, , } MRP(c) =
| FNG(c) ∩P | | TPG(c) ∩P | + | FNG(c) ∩P |
C = { arg max FPPI(c) ∣ ∈ }
FPPI(c)≤ : (7) (8) (9) H(c) =
GDPI() =
with = {10 −2, 10−1.78, … , 100} and | | = 9 .
Ghost Detections per Image Since not all false positives are equally disruptive, we propose to focus on the number of ghost detections per image 9, we denote the set of confidence levels for ghost detections as Cℋ. The filtered log-average miss rate with respect to ghost detections (FLAMRPℋ) is defined as: 1 | H(c) |. Based on GDPI() and Equation
1 | Cℋ |
∈ Cℋ FLAMRPℋ = exp ( ∑ log MRP()) (10)
From a safety perspective, we are interested in finding an operating point for a PD where no safety-critical pedestrian is missed. It is still open to what extent this requirement can be relaxed for DNNs for object tracking. In this work, we are focused on single images and define a safety-critical pedestrian as any pedestrian who is in the foreground. Furthermore, we assign the operating point to a confidence threshold that must be determined post-hoc to PD training. We define the confidence threshold ∈ [0, ℱ∗] and the upper bound ℱ∗ as the confidence threshold with the lowest miss rate for foreground ℱ: ℱ∗ = arg min MRℱ(). If MRℱ( ℱ∗) = 0holds for a given validation dataset, it can be ensured that every safetycritical pedestrian in the foreground is correctly detected. Lowering the confidence threshold so that < ℱ∗ may improve performance for other error categories, but is not capable of causing foreground errors. Consequently, we see ℱ∗ as a reasonable choice for an operating point. The corresponding amount of ghost detections per image is given by GDPI( ℱ∗).
In total, we trained and analyze results for 44 PDs i.e. 11 PDs for each of the four diferent feature extractions and backbones. PDs with the same feature extraction also difer since the randomly initialized AL parameters in the feature extraction and perception heads change for every training run. We report confidence intervals ( 0.95), using a student’s t-distribution due to the small sample size. This accounts for randomness and improves transparency, although satisfactory sample sizes are dificult when working with large DNNs.
10 RP M8 A L F 6 4 2 08 9 LAMR scores for the reasonable subset of the CityPersons validation dataset are reported in Table 3. BGC-HRNet- Figure 6: LAMR scores for the reasonable subset compared w32 has the best LAMR performance, which confirms tgohothstedfieltteecretidonlosg(-FaLvAerMagReP,mFiLsAsMraRtePℋw).ith and w/o respect to the reported benchmarks listed in Table 1. In summary, our experiments show overlapping 0.95, indicating that randomness in initialization and training influences performance. Interestingly, our PD with BGC-HRNet-w32 reveal that performance of FPN-ResNet-50 is comparaas backbone achieves a score very similar to the results ble by achieving the same lower bound of 0.95. This reported for BGCNet [17] despite using a simpler archi- is contradictory to the fact that BGC-HRNet-w32 outtecture that does not employ box-guided convolutions. performs FPN-ResNet-50 by nearly 2% in LAMRr. We observe the reversed efect for MDLA-UP-34 which per5.2. Bias of Reasonable Subset forms second-best in LAMRr but achieves the worst result for FLAMRℋ.
ℱ Compared to the LAMRr scores on the reasonable subset, Figure 6 shows how the proposed focus on ghost dewe see a corresponding order of the FLAMRP scores for tections almost resolves the weak dependence between background in Table 3. This indicates that the reason- FLAMRPℋ to LAMRr in the foreground. Hence, the able subset holds a vast amount of smaller pedestrians in FLAMRPℋ efectively measures performance diferently the background. BGC-HRNet-w32 performs best for all and considers factors that are ignored by the LAMRr. performance metrics and subsets. In contrast, FLAMRℱ The results show that PDs optimized for LAMRr do not scores contradict the LAMRr results with a diferent rank- necessarily perform best with respect to FLAMRP or ing of PDs and strongly overlapping 0.95. The inherent FLAMRPℋ. Controversially, there are PDs (with FPNbias of the LAMRr evaluation leads to underestimation ResNet-50) that have a lower FLAMRℱℋ score despite a of the true potential of certain feature extractions and much higher LAMRr score. These models have a lower backbones for the highly safety-critical foreground cate- miss rate for pedestrians in the foreground and produce gory. fewer ghost detections per image. Thus, the LAMRr for
Figure 6 analyzes the dependence of LAMRr scores the reasonable subset has limits in terms of an applicationand FLAMRP and FLAMRPℋ for foreground and back- oriented evaluation. The problem arises from the training ground. Whereas FLAMRP scores are strongly correlated strategy for PDs that is not focused on safety. The sewith LAMRr scores in the background, the dependence lection of the best performing checkpoint in terms of in the foreground is lower. Our analysis shows that the the LAMRr is disconnected from the evaluation of safetyreasonable subset is dominated by pedestrians in the critical pedestrians. background, which are less safety-critical. Based on the large deviations between FLAMRPℋ to LAMRr, we conclude that FLAMRPℋ introduces a new 5.3. Application-oriented Evaluation application-oriented perspective for the evaluation of DNNs for pedestrian detection. This conclusion seems reasonable due to the systematic categorization of errors. Here, safety-critical pedestrians are identified as the complement of highly occluded pedestrians and distant pedestrians.
Evaluating the miss rate in foreground ℱ and background ℬ while only considering ghost detections per image (GDPI) combines the evaluation of opposed critical cases: Missing a safety-critical pedestrian or predicting non-existing pedestrians. The evaluation of the best run in terms of LAMRr and FLAMRP in Table 3 shows BGC-HRNet-w32 as the far superior choice. However, FLAMRℱℋ-scores based on our systematic error categories
Towards an application-oriented analysis of DNNs for pedestrian detection, we determine upper bounds on the confidence threshold ℱ∗ as operating points for individual FLAMRP and FLAMRPℋ average over multiple confidence PDs. Results can be seen in Figure 7. We see that between thresholds and put less weight on safety-relevant ranges training runs of PDs with the same feature extraction, towards ℱ∗. the upper bound of the confidence threshold ℱ∗ and the required GDPI( ℱ∗)vary greatly. Thus, operating points 100 must be determined individually for the PDs and cannot be specified in general for a particular feature extraction.
FPN-ResNet-50-1 FPN-ResNet-50-2 FPN-ResNet-50-3 BGC-HRNet-w32-1 BGC-HRNet-w32-2 BGC-HRNet-w32-3 FPPI values ()c R M
FPN-ResNet-50-1 FPN-ResNet-50-2 FPN-ResNet-50-3 BGC-HRNet-w32-1 BGC-HRNet-w32-2 BGC-HRNet-w32-3
GDPI values 10 1 )(c R
M10 2
Furthermore, our evaluation shows that not every PD is capable of detecting every pedestrian in the foreground with a confidence threshold of 0.01. This means that there are PDs with MRℱ(0.01) ≠ 0(CSP-ResNet-50: 2, FPN-ResNet-50: 4, MDLA-UP-34: 0, BGC-HRNet-w32: 1). In general, foreground pedestrians are missed with a maximum of MRℱ( ℱ∗) = 0.29%. Up to this point, the subset-based evaluation of DNNs for pedestrian detection has largely focused on benchmarking. Due to the limited informative value, it was dificult to derive guidelines for the application-oriented development process of DNNs.
We take the stance that aggregated performance metrics such as FLAMRℋ must be collated with metrics such as
P MRℱ( ℱ∗)and GDPI( ℱ∗).
In this work, we propose a rule-based error categoriza1125..50 LMAeMtrRicr tdieotnectotioevna. luMatueltthipelepedrifsojrominatncceatoefgaorDieNsNfoforrfpaelsdeesntreigaanitrceM1075...050 FFFCeLPSaANPtM--uRRrReeessENNxeetttr--a55c00tion itnivetshearfeodreefingeroduinndo.rdTerhteodidisetninticftyiosnafeistyb-carsietidcaol nertrhorrsee 2.5 BMGDCL-AH-RUNP-e3t4-w32 occlusion-related categories and the braking distance of 0.00.00 0.25 0.50 0.75 1.G0D0PI(c*1).25 1.50 1.75 2.00 an automated driving system. We expect that the inclusion of depth information would improve the separation Figure 8: LAMRr and FLAMRPℋ are independent of GDPI. between foreground, background, occluding pedestrians and environment. In future work, we would like to reevaluate the performance of DNNs specifically designed for
Figure 8 shows that performance metrics (LAMRr the occlusion problem using our proposed error cateand FLAMRPℋ) are unrelated to the required number of gories. We identify three categories of false positives, ghost detections GDPI( ℱ∗). Surprisingly, FPN-ResNet-50 with ghost detections being the most disruptive. For our achieves with 0.48 the lowest value of GDPI( ℱ∗). The rea- experiments, we use a simple and generic framework to son for the unexpected behavior can be seen in Figure 9. build DNNs for pedestrian detection. In consequence, Although the sorted miss rate curves of the selected PDs we train 44 DNNs based on four commonly used backare close in the middle range, they diverge the most in the bones, achieving state-of-the-art performance in terms head and tail ranges. The vertical lines mark the common of LAMRr. The goal of our application-oriented evaluvalues for which the miss rate is averaged. As a conse- ation is two-folded. To account for safety-critical false quence, aggregated performance metrics such as LAMRr, negatives as well as disruptive false positives, we propose FLAMRℱℋ as a new performance metric. Finally, we determine an operating point as the confidence threshold where no pedestrian in the foreground is missed. By revisiting and refining the current evaluation, we contribute to a safety-focused development process of DNNs for pedestrian detection.
The research leading to these results is funded by the German Federal Ministry for Economic Afairs and Energy within the project “Methoden und Maßnahmen zur Absicherung von KI basierten Wahrnehmungsfunktionen für das automatisierte Fahren (KI-Absicherung)”. The authors would like to thank the consortium for the successful cooperation. The authors acknowledge support by the state of Baden-Württemberg through bwHPC. The cityscapes dataset for semantic urban scene understanding, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3213–3223. [24] F. Kraus, N. Scheiner, W. Ritter, K. Dietmayer, Using machine learning to detect ghost images in automotive radar, in: IEEE Conference on Intelligent Transportation Systems (ITSC), 2020, pp. 1–7.