Domain-centric ADAS datasets

Abstract

Since the rise of Deep Learning methods in the automotive field, multiple initiatives have been collecting datasets in order to train neural networks on diferent levels of autonomous driving. This requires collecting relevant data and precisely annotating objects, which should represent uniformly distributed features for each specific use case. In this paper, we analyze several large-scale autonomous driving datasets with 2D and 3D annotations in regard to their statistics of appearance and their suitability for training robust object detection neural networks. We discovered that despite spending huge efort on driving hundreds of hours in diferent regions of the world, merely any focus is spent on analyzing the quality of the collected data, from an operational domain perspective. The analysis of safety-relevant aspects of autonomous driving functions, in particular trajectory planning with relation to time-to-collision feature, showed that most datasets lack annotated objects at further distances and that the distributions of bounding boxes and object positions are unbalanced. We therefore propose a set of rules which help find objects or scenes with inconsistent annotation styles. Lastly, we questioned the relevance of mean Average Precision (mAP) without relation to the object size or distance.

Keywords

• eol>Advanced Driver-Assistance Systems
• Trajectory Planning
• Domain-centric Datasets
• Object Detection
• mean Average Precision

1. Introduction and Motivation

SafeAI 2023: The AAAI’s Workshop on Artificial Intelligence Safety, Feb 13-14, 2023 Washington, D.C., US * Corresponding main author. $ [e-mail] (V. Diviš); [e-mail] (T. Schuster); [e-mail] (M. Hrúz) € https://gitlab.com/divisvaclav/ (V. Diviš) 0000-0001-9935-7824 (V. Diviš); 0000-0002-9421-8566 (M. Hrúz)

© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) 2.1. Automotive Datasets • We define a minimum safe distance = areas whereas A2D2 and ONCE also contain highways, (, ) for a variety of scenarios (ego speed and country roads, tunnels etc. On top of that, the sensor weather dependent deceleration ). We calculate setups vary as well, e.g. diferent camera resolutions. For the safe distance based on German legislation, but the the KITTI dataset, the authors used a LIDAR sensor and process can easily be adapted to any other legislation. two stereo cameras (left and right), whereas for Waymo ifve LIDARs (restricted to 75m) and five cameras were • We analyze the distribution of objects’ bounding boxes’ used. nuScenes used six cameras and one LIDAR sensor (BB) relative sizes, distances to ego vehicle, and posi- as well as five radars, ONCE uses one LIDAR and seven tions in the datasets. cameras, and A2D2 five LIDARs and six cameras. How• We define a set of standardizable sanity checkers which ever, for the ACC, only the front cameras are taken into help verify the quality of the collected data and mark account, resulting in a smaller amount of usable images. ambiguously labeled data. The KITTI dataset is the smallest in terms of scenes and the least diverse, containing only sunny and cloudy • We highlight the concrete missing information which daytime scenes. For a short period of time, the Waymo is not part of the datasets and diagnose the cause. and nuScenes datasets provided the largest variety and amount of data and annotations; they are among the most • We propose an automotive mean Average Precision widely-used autonomous driving datasets. Although the (amAP) metric, which is now related to the distance to ONCE dataset recently set a new benchmark for the the object, or relative BB size. amount of driving hours and frames, Waymo contains the highest amount of 3D bounding boxes because ONCE 2. Datasets and Related Work focuses on self-supervised learning without labels. Table 1 gives an overview of important general information per dataset.

As mentioned in Section 1, the prerequisite for the cor- 2.2. Dataset Analysis rect functionality of the ACC is information about object classes, sizes (used in case of overtaking or evasive The authors of the above-mentioned datasets compared maneuver), and distances of the objects to the ego ve- their works based on the common aspects of the datasets hicle. Several common automotive datasets are there- as shown in Table 1. General properties like the number fore not suitable for such a task, despite some of them of driving hours are often used to compare datasets and to providing LIDAR data, namely BDD100K [ 9 ] (no LIDAR state an improvement. Moreover, the number of scenes, data), CityScapes [ 10 ] (no bounding box annotations), images, or annotations is often used to determine the Perl [ 11 ] (no 3D annotations) or Apollo Scape [12] (no quality of the datasets. images, only LIDAR). The other group of datasets con- For instance, the authors of the ONCE [ 18 ] and tains images and LIDAR point clouds including the 2D A2D2 [14] dataset focus on the number of annotations, as well as the 3D annotations. It is for this reason that the amount of driving hours, the adverse weather condiwe decided to take the following large-scale automotive tions, the time (day/night) and diferent locations (urban, datasets into account: KITTI [13], Audi Autonomous highway, country roads) as well as countries/cities where Driving Dataset (A2D2) [14], Lyft Level 5 dataset [15], the data was captured. However, specific requirements nuScenes dataset [ 16 ], Waymo Open Dataset [ 17 ] and of driver assistance systems were not considered while the ONCE dataset [ 18 ]. creating or evaluating any of those datasets. The intent

Each dataset contains a various number of labeled was rather to generate general datasets for a wide range objects like small vehicles (vehicle, ego vehicle, SUV, mo- of supervised and unsupervised learning tasks as well as torcycle, etc.), large vehicles (truck, bus, tram), pedestri- driving functions. ans, and cyclists. Especially KITTI and nuScenes show The authors of A2D2, as well as nuScenes, focused high-class imbalance for some classes due to fine-grained on statistics relevant to the ACC and other assistance classes. Furthermore, the selected datasets contain la- systems. Their work provides information about the beled camera images with 2D and 3D bounding boxes and distribution of the object distances for diferent classes the corresponding LIDAR point cloud information which as well as the absolute number of objects within the provides distance information for each object. However, dataset. Additionally, the authors of nuScenes analyzed the size of the datasets, in terms of the number of labeled the distributions of the velocities of common objects like frames and captured ambient conditions, varies. For in- vehicles and bikes as well as bounding box dimensions. stance, A2D2 provides a dataset of 2D labeled images, but The KITTI benchmark [13] is based on the perforonly a small part contains 3D bounding boxes. KITTI, mance analysis of neural networks on the size of BBs nuScenes, LyftLevel5 and Waymo reflect only urban in pixels as a proxy for the distance of the ego vehicle to the object. Their work in general follows the COCO on the same path". Let’s define the first object to be an evaluation methodology [ 19 ], but no physical distance in- obstacle (anything else than the ego vehicle) and the formation is used. The authors of nuScenes and Waymo second object to be the ego vehicle. We consider the vedid set a baseline for various detection tasks, yet without locity of an obstacle to be equal to 0km/h (representing considering the distances to the diferent objects explic- the stand-still object and therefore the worst-case sceitly. The analysis closest to ours is done by the authors nario), and the ego vehicle’s deceleration to be 7 /2 of the ONCE dataset. They analyzed the collected data (can variate within a range from 7 /2 till 10 /2 on regarding distance-wise mean Average Precision perfor- dry roads [ 21 ],[ 22 ]). Vehicle deceleration can be seen as mance for 3D object detection using only point clouds. a function of adhesion between the tires and the road, However, their distance thresholds were selected rather which depends on the material used in the tires, material intuitively, whereas we specifically derive the distance of the road, temperature, weather conditions, mounted from the domain safety requirements. Additionally, we braking system, and the mass of the vehicle. analyze the spatial distribution of objects within the im- Based on the definition of TTC, let us consider three ages as well as the bounding box/object size compared driving scenarios: to the image size. • highway (recommended speed 130/ℎ ≈ 36/) • country road (maximum speed 100/ℎ ≈ 28/)

3. Background

• city (maximum speed 50/ℎ ≈ 14/)

Since the majority of related works only analyze datasets

from a general ML perspective, omitting the point of We now compute the minimal safe distance which data-centric paradigm, we decided to verify the SOTA au- needs to be ensured in order to brake in time (withtomotive datasets in regard to a trajectory planning task. out initiating any evasive maneuver), for the following One part of our motivation is that forecasting the trajec- case: ego vehicle is driving on the highway, the postory planning is conditioned by ego’s vehicle velocity, sible deceleration is equal to 7/2 and reaction dewhich in case of higher value takes the further-distance lay is 0.0. Based on the kinematic equations of a linobjects into account. In order to be able to evaluate the early decelerating object, the distance which the object sduafictaiesnetcsy, waned’vqeucahliotyseonf tahnenToitmatee-dToob-Cjeocltlsiswiointhaisn athnein- wwihllerteratvimeleisaisfuanfcutinocntioofntoimfedece=lera0ti+on0 =− 210−2 ,. stance to calculate the minimal safe distance from the With a linear deceleration of 7/2, the vehicle, moving ego vehicle. within the legal limits, will reach its standstill state in obsFtoarcltehheesaadkinegoffrosmimtphleicoitpyp, owsieteddoirencottiocnon(osnidtehreacnoyl- ftrimame e, t=he e0g−o v=ehic3l6e7− w0il=ltr5a.v1e4l.a Wdisitthanincethoifs tim=e lision course), since we are working with static images 0 + 36 · 5.14 − 12 · 7 · 5.142 = 185.04 − 92.46 = 92.58. and thus don’t have the information about the relative For completeness, the braking distance under the same motion of the objects. As described in [ 20 ]: "The TTC weather conditions on country roads is 55.11 and in value at instant is defined as the time for two objects the city 14 as can be seen in Table 2. As mentioned to collide if they continue at their present velocity and earlier, this process can be generalized and repeated for rather generally (small, middle, big), we propose to have clearly specified operational domain dependencies and incorporate the minimum safe distances as thresholds.

Furthermore, the relation of AP to the objects’ relative BB sizes or distance highlights detailed discrepancies between the model’s performance trained on the same dataset. We therefore incorporate the minimal safe distance of each scenario as a threshold for the creation of test subsets ⊂ ℬ ⊂ from original test dataset . For instance contains objects only in distance > (ℎℎ). For each subset the average precision can be calculated, representing a concrete value for a specific operational domain e.g. driving in the city. Equation 1 represents a diferent perspective on the average precision metric, which we call automotive mean AP. any ambient conditions, type of vehicle, and speed limitations.

It is noticeable that the highway’s maximum foresight boundary will be in reality limited by the physical properties of the camera or the maximum speed diference between the ego vehicle and the object. But most impor- tantly, objects within those safe ranges must be part of = 1 ∑︁ , (1) a dataset (training and testing), otherwise, the system =1 will deal with an epistemic uncertainty [ 23 ]. In order where is a number of domain specific test-subsets. to be able to investigate the statistics of objects’ appearances, we need a dataset that provides information about the object’s distance. As mentioned in Section 2, some 4. Analysis of datasets datasets such as Perl and BKK100, etc. are not suitable for this task. Consequently, we have chosen the following In this chapter we analyze all datasets from Table 1 for large-scale automotive datasets: KITTI, Waymo open the characteristics mentioned in Section 3. In order to Dataset, A2D2 from Audi, nuScenes, LyftLevel5 and evaluate the model’s generalization ability, the collected ONCE, which contain 3D annotations and distances to data has to be divided into two parts, namely training the objects. and validation. The training part is used for extracting

In regard to the functionality of trajectory planning, we the relevant features and finding a reasonable combinafocused on the following in-dataset object characteristics: tion in order to build a high-level feature representation, whereas the validation part is used to evaluate the loss • distribution of a BB’s relative size: in order to verify after each training cycle (epoch). The training loop usuthat a variety of objects’ sizes is captured within the ally ends when the loss, on a validation set, stagnates for dataset, several epochs [ 24 ]. Logic demands that both the training and validation parts should have uniformly distributed • distribution of the distance between obstacles and the objects densities and their properties (e.g.size, distance). ego vehicle (with relation to minimum safe distances): We therefore decided to evaluate the correlation between in order to verify that objects in further distances are the theoretical uniform distribution and observed one by incorporated within the dataset, calculating the Wasserstein distance [ 25 ]. • relation between BB’s relative size and object distance from ego vehicle: to discover abnormality within the dependency, • heatmap of an object’s appearance density: to visualize the potential asymmetrical appearance of an object with relation to the ego vehicle, • an optical flow between consecutive images; in order to identify a series of static images, which can lead to class imbalanced dataset.

As originally presented in [19] and further explored in [18], it seems to be reasonable to observe the mean Average Precision with relation to a specific object’s size. Since the original authors clustered the object groups

4.1. Analysis of outliers There is no reason to assume that all data are flawlessly annotated since the task is usually done by several people, which increases the uncertainty of an inconsistent annotation style. We therefore designed plausibility check functions which allow us to indicate the potentially wrong annotated data and analyze them later on. • 1 : (0.0 ≥ .. > 1.0) • 2 : (− +.. )

• 3 : ((, ) > ℎ) • 4 : (. (− 1, ) > ℎ) 7 w lo6 lf cap it5 o s4 e n e d f3 o e d itu2 n g a1 M 0

Exemplary results of the analysis of the objects’ distances

distribution as well as the relation between the relative BB size and object distance can be seen in Figures 3 and 4. Moreover, we show an example of object appearance variation (heatmap) of class Vehicle in the A2D2’s dataset in Figure 5.

The function 1 returns a bounding box whose rela

tive size is out of the range (0.0, 1.0]. The outliers of otherwise exponentially decayed objects’ size with respect to the distance to the ego are marked by function 2. This function covers the quadratic dependency of the BB area and the mapping of the captured object within the real world to the pixel coordinates. The parameter .. and the denominator were found with the least squares optimization from the analyzed data and therefore are unique for each dataset and each class. The function 3 highlights objects whose bounding boxes significantly overlap. The threshold of the relative intersection area can be defined by ℎ.

Examples of the outcomes for functions 1 − 3 are given in Figure 1. We further found that most of the datasets contain sequences of "stop and go" in a trafic jam, or idling at crossroads. These situations result in the recording of many similar images without objects or surrounding variation. Therefore we added 4, which calculates a dense optical flow [ 26 ] from previous and actual images in the sequence and returns a positive flag in case the magnitude sinks bellow empirically defined threshold as seen in Figure 2. The higher the value of the magnitude, the more objects were moving from frame to frame. With this method, we could even identify if the data were recorded repeatedly in the same place [ 27 ] (when recorded in one session), but we only used it to discover static scenarios.

(a) (c) (b) (d) Minimum safe distance in [m] for 130km/h for 100km/h for 50km/h 80 100 120 20 40

60

Distance to object [m]

5. Concluding Remarks

the annotation style (objects under a certain pixel area were excluded from the annotation process), and the ambient conditions in which the dataset was recorded. In addition, a system trained on such a dataset would have to deal with epistemic uncertainty and look for additional sources of information (namely LIDAR).

Furthermore, all datasets contain predominantly smallsized objects (the highest MEAN value of the relative BB size of the class Person was 0.091 in the case of the KITTI dataset). For comparison, the same can be stated for the well-known COCO dataset [ 19 ], where the class Person has a MEAN relative BB size equal to 0.089. By generating heatmaps, we discovered that 99.8% of the objects appear only in the two lower quadrants of the image. Such information can lead to a significant downsizing of the ifeld of view and the thereof acceleration of the detectors’ inference time. The majority of overlapping BBs, with potentially wrong annotation styles, were extracted from a sequence of streams on crossroads. Such a static data sequence (9.55% in case of nuScenes dataset) contains a lot of similar features (the majority of surrounding objects are not moving) and could be removed from the dataset.

Finally, we defined and evaluated a reasonable set of rules, described in Section 4.1, which automatically proves the quality of the collected data from a domainrelated perspective. We encourage the community to use our "domain-centric" approach in order to create a dataset under concrete functional constraints and train detectors on it. Our code and additional results are published on GitLab and can be publicly accessed. 1

This work deepened our vision of a domain-centric ML approach in the automotive industry. To conclude, we outline some research directions which we are currently investigating: (a) Analysis of automotive mAP based on 1https://gitlab.com/arrk-fi/ObjectDetectionCriticality/-/tree/ dependency_branch. 1.0 0.8 itsy end0.6 cyen uq0.4 e fr 0.2 0.0 0.0

0.2 Distribution of relative BB sizes for objects: Small_Vehicle

A2D2 nsamp =25043, wdist =0.96 Kitti nsamp =28742, wdist =0.98 ONCE nsamp =33292, wdist =0.98 nuScenes nsamp =139575, wdist =0.99 Waymo nsamp =1280729, wdist =0.99 LyftLevel5 nsamp =190029, wdist =0.99 Distribution of distances for objects: Pedestrian

LyftLevel5 nsamp =8751, wdist =0.60 nuScenes nsamp =2686, wdist =0.82 Waymo nsamp =767542, wdist =0.65 A2D2 nsamp =4388, wdist =0.80 ONCE nsamp =3006, wdist =0.86 Kitti nsamp =4487, wdist =0.89 1.0

Acknowledgments

This work is partly funded by ARRK Engineering GmbH.

The work has also been supported by the grant of the University of West Bohemia, project No. SGS-2022-017 and by the Technology Agency of the Czech Republic, project No. CK03000179. the relative BB size or distance to object with SOTA object detectors. (b) Object detector performance analysis on cleaned data (without outliers). (c) Dataset creation with respect to our domain-centric approach. (d) Combination of datasets in order to achieve a more uniform data distribution. (e) Data augmentation to compensate weak aspects in the datasets.

References

[7] SAE International , Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems , volume J3016 , 2014 .

[8] A. Gasparetto , P. Boscariol , A. Lanzutti , R. Vidoni , Path planning and trajectory planning algorithms: A general overview, Motion and operation planning of robotic systems ( 2015 ) 3 - 27 .

[9] F. Yu , W. Xian , Y. Chen , F. Liu , M. Liao , V. Madhavan , T. Darrell, BDD100K: A diverse driving video database with scalable annotation tooling , CoRR abs/ 1805 .04687 ( 2018 ). URL: http://arxiv.org/ abs/ 1805 .04687. arXiv: 1805 .04687.

[10] M. Cordts , M. Omran , S. Ramos , T. Rehfeld , M. Enzweiler , R. Benenson , U. Franke , S. Roth , B. Schiele , The cityscapes dataset for semantic urban scene understanding , in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016 , pp. 3213 - 3223 .

[11] G. Pandey , J. R. McBride , R. M. Eustice , Ford campus

[1] E. Schwalb , Analysis of safety of the intended use vision and lidar data set , The International Journal (sotif) ( 2019 ). of Robotics Research 30 ( 2011 ) 1543 - 1552 .

[2] W. W. Royce , Managing the development of large [12] Y. Ma , X. Zhu , S. Zhang , R. Yang , W. Wang, software systems: concepts and techniques , in: D. Manocha, Traficpredict: Trajectory prediction Proceedings of the 9th international conference on for heterogeneous trafic-agents , in: Proceedings Software Engineering , 1987 , pp. 328 - 338 . of the AAAI Conference on Artificial Intelligence,

[3] K. Petersen , C. Wohlin , D. Baca , The waterfall volume 33 , 2019 , pp. 6120 - 6127 . model in large-scale development , in: International [13] A. Geiger , P. Lenz , C. Stiller , R. Urtasun , Vision Conference on Product-Focused Software Process meets robotics: The kitti dataset , The International Improvement , Springer, 2009 , pp. 386 - 400 . Journal of Robotics Research 32 ( 2013 ) 1231 - 1237 .

[4] ISO 26262 : 2018 , Road vehicles - Functional safety [14] J. Geyer , Y. Kassahun , M. Mahmudi , X. Ricou , (ISO 26262) , Standard, International Organization R. Durgesh , A. S. Chung , L. Hauswald , V. H. Pham , for Standardization, 2018 . M. Mühlegg , S. Dorn , T. Fernandez , M. Jänicke,

[5] ISO/PAS 21448: 2019 , Road vehicles - Safety of the S . Mirashi, C. Savani , M. Sturm , O. Vorobiov , intended functionality, Standard, International Or- M. Oelker , S. Garreis , P. Schuberth, A2d2: Audi auganization for Standardization, 2019 . tonomous driving dataset, 2020 . URL: https://arxiv.

[6] A. Ng , Deep Learning AI data-centric ai com - org/abs/ 2004 .06320. doi: 10 .48550/ARXIV. 2004 . petition, https://https-deeplearning-ai. github.io/ 06320 . data-centric-comp/, 2021 . Accessed: 2022 -08-16. [15] R. Kesten , M. Usman , J. Houston, T. Pandya, K. Nadhamuni , A. Ferreira , M. Yuan , B. Low , A. Jain , P. Ondruska , S. Omari , S. Shah , A. Kulkarni , A. Kazakova , C. Tao , L. Platinsky , W. Jiang , V. Shet , Level 5 perception dataset 2020 , https://level-5.global/level5/data/, 2019 .

[16] H. Caesar , V. Bankiti , A. H. Lang , S. Vora , V. E. Liong , Q. Xu , A. Krishnan , Y. Pan , G. Baldan , O. Beijbom, nuscenes: A multimodal dataset for autonomous driving , in: CVPR , 2020 , pp. 11621 - 11631 .

[17] P. Sun , H. Kretzschmar , X. Dotiwalla , A. Chouard , V. Patnaik , P. Tsui , J. Guo , Y. Zhou , Y. Chai , B. Caine , et al., Scalability in perception for autonomous driving: Waymo open dataset , in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020 , pp. 2446 - 2454 .

[18] J. Mao , M. Niu , C. Jiang , X. Liang , Y. Li , C. Ye , W. Zhang , Z. Li , J. Yu , C. Xu , et al., One million scenes for autonomous driving: Once dataset , 2021 .

[19] T.-Y. Lin , M. Maire , S. Belongie , J. Hays , P. Perona , D. Ramanan , P. Dollár , C. L. Zitnick , Microsoft coco: Common objects in context , in: ECCV, Springer, 2014 , pp. 740 - 755 .

[20] R. Van Der Horst , J. Hogema, Time-to-collision and collision avoidance systems ( 1993 ).

[21] I. für Unfallanalysen , Bremstabelle hamburg - institut für unfallanalysen - bremstabelle , https://unfallanalyse.hamburg/index.php/ ifu-lexikon/bremsen/bremstabelle-a/, 2022 . Accessed: 2022 -09-14.

[22] A. Erd , M. Jaśkiewicz , G. Koralewski , D. Rutkowski , J. Stokłosa , Experimental research of efectiveness of brakes in passenger cars under selected conditions , in: 2018 Xi International Science-Technical Conference Automotive Safety, IEEE , 2018 , pp. 1 - 5 .

[23] A. Kendall , Y. Gal , What uncertainties do we need in bayesian deep learning for computer vision? , in: NIPS, 2017 , pp. 5574 - 5584 .

[24] L. Prechelt , Early stopping-but when? , in: Neural Networks: Tricks of the trade , Springer, 1998 , pp. 55 - 69 .

[25] C. Villani , Optimal transport, old and new. notes for the 2005 saint-flour summer school , Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer 3 ( 2008 ).

[26] G. Farnebäck , Two-frame motion estimation based on polynomial expansion , in: Scandinavian conference on Image analysis , Springer, 2003 , pp. 363 - 370 .

[27] A. J. Davison , Real-time simultaneous localisation and mapping with a single camera , in: Computer Vision , IEEE International Conference on, volume 3 , IEEE Computer Society, 2003 , pp. 1403 - 1403 .