The shift from classical programming paradigms to Machine Learning-driven approaches (ML) is significant. Humans are nowadays often relying on decisions and recommendations made by Artificial Neural Networks (ANNs), however most of the applications are harmless. This is not the case of Advanced Driver-Assistance Systems (ADAS), where wrong decisions can lead to severe injuries [1]. The development of an ML-driven application within this field follows strict processes [2,3], from acquiring data necessary for training to deploying and evaluating the models. These processes comply with functional safety standards [4,5] but do not propose specific measures, nor concrete thresholds which the system should pass before being publicly released. Moreover, it is generally not feasible to collect the full variation of information in a stochastic environment such as public roads. This is why it is important to look deeper into the possibilities of diagnosing and analyzing unbalanced or missing information within large-scale datasets related to the operation domain, i.e. a data-centric approach [6].
A data-centric approach lays the focus on ensuring that the data clearly conveys what the AI must learn. In the case of higher levels of ADAS [7], the vehicle can control the movement in longitudinal and lateral directions, while relying predominantly on cameras, radars, and nowadays more frequently on LIDAR systems. One example of an ADAS function is the Adaptive Cruise Control (ACC). The task of ACC is to drive within the lane at a certain speed and keep a safe distance from any potential obstacle. The Time-To-Collision (TTC) for each object is calculated for the purpose of keeping a safe distance. If the TTC value decreases below a defined threshold, a braking or evasive maneuver will be initiated. Based on linear kinematic equations, where distance is equal to speed over time, the greater the speed difference between the subject-automated vehicle (ego vehicle) and the object in front of it, the shorter the TTC will be, hence the ACC needs to plan in advance. As a consequence, on highways, the ACC must incorporate objects at a further distance (on camera images objects will appear smaller) into the trajectory planning [8]. On the contrary, in cities, where the maximum speed is limited to 50km/h, the average area of an encountered object will be larger. Motivated by these physical dependencies, we took a deeper look at the annotated objects and analyzed their statistical appearance. The examination was done in regard to the functionality of ACC, in state-of-the-art (SOTA) large-scale automotive datasets. Our contributions are as follows:
• We define a minimum safe distance 𝑠𝑚𝑖𝑛 = 𝑓 (𝑣, 𝑎𝑚𝑎𝑥) for a variety of scenarios (ego speed 𝑣 and weather dependent deceleration 𝑎𝑚𝑎𝑥). We calculate the safe distance based on German legislation, but the process can easily be adapted to any other legislation.
• We analyze the distribution of objects' bounding boxes' (BB) relative sizes, distances to ego vehicle, and positions in the datasets.
• We define a set of standardizable sanity checkers which help verify the quality of the collected data and mark ambiguously labeled data.
• We highlight the concrete missing information which is not part of the datasets and diagnose the cause.
• We propose an automotive mean Average Precision (amAP) metric, which is now related to the distance to the object, or relative BB size.
As mentioned in Section 1, the prerequisite for the correct functionality of the ACC is information about object classes, sizes (used in case of overtaking or evasive maneuver), and distances of the objects to the ego vehicle. Several common automotive datasets are therefore not suitable for such a task, despite some of them providing LIDAR data, namely BDD100K [9] (no LIDAR data), CityScapes [10] (no bounding box annotations), Perl [11] (no 3D annotations) or Apollo Scape [12] (no images, only LIDAR). The other group of datasets contains images and LIDAR point clouds including the 2D as well as the 3D annotations. It is for this reason that we decided to take the following large-scale automotive datasets into account: KITTI [13], Audi Autonomous Driving Dataset (A2D2) [14], Lyft Level 5 dataset [15], nuScenes dataset [16], Waymo Open Dataset [17] and the ONCE dataset [18]. Each dataset contains a various number of labeled objects like small vehicles (vehicle, ego vehicle, SUV, motorcycle, etc.), large vehicles (truck, bus, tram), pedestrians, and cyclists. Especially KITTI and nuScenes show high-class imbalance for some classes due to fine-grained classes. Furthermore, the selected datasets contain labeled camera images with 2D and 3D bounding boxes and the corresponding LIDAR point cloud information which provides distance information for each object. However, the size of the datasets, in terms of the number of labeled frames and captured ambient conditions, varies. For instance, A2D2 provides a dataset of 2D labeled images, but only a small part contains 3D bounding boxes. KITTI, nuScenes, LyftLevel5 and Waymo reflect only urban areas whereas A2D2 and ONCE also contain highways, country roads, tunnels etc. On top of that, the sensor setups vary as well, e.g. different camera resolutions. For the KITTI dataset, the authors used a LIDAR sensor and two stereo cameras (left and right), whereas for Waymo five LIDARs (restricted to 75m) and five cameras were used. nuScenes used six cameras and one LIDAR sensor as well as five radars, ONCE uses one LIDAR and seven cameras, and A2D2 five LIDARs and six cameras. However, for the ACC, only the front cameras are taken into account, resulting in a smaller amount of usable images.
The KITTI dataset is the smallest in terms of scenes and the least diverse, containing only sunny and cloudy daytime scenes. For a short period of time, the Waymo and nuScenes datasets provided the largest variety and amount of data and annotations; they are among the most widely-used autonomous driving datasets. Although the ONCE dataset recently set a new benchmark for the amount of driving hours and frames, Waymo contains the highest amount of 3D bounding boxes because ONCE focuses on self-supervised learning without labels. Table 1 gives an overview of important general information per dataset.
The authors of the above-mentioned datasets compared their works based on the common aspects of the datasets as shown in Table 1. General properties like the number of driving hours are often used to compare datasets and to state an improvement. Moreover, the number of scenes, images, or annotations is often used to determine the quality of the datasets.
For instance, the authors of the ONCE [18] and A2D2 [14] dataset focus on the number of annotations, the amount of driving hours, the adverse weather conditions, the time (day/night) and different locations (urban, highway, country roads) as well as countries/cities where the data was captured. However, specific requirements of driver assistance systems were not considered while creating or evaluating any of those datasets. The intent was rather to generate general datasets for a wide range of supervised and unsupervised learning tasks as well as driving functions.
The authors of A2D2, as well as nuScenes, focused on statistics relevant to the ACC and other assistance systems. Their work provides information about the distribution of the object distances for different classes as well as the absolute number of objects within the dataset. Additionally, the authors of nuScenes analyzed the distributions of the velocities of common objects like vehicles and bikes as well as bounding box dimensions.
The KITTI benchmark [13] is based on the performance analysis of neural networks on the size of BBs in pixels as a proxy for the distance of the ego vehicle to the object. Their work in general follows the COCO evaluation methodology [19], but no physical distance information is used. The authors of nuScenes and Waymo did set a baseline for various detection tasks, yet without considering the distances to the different objects explicitly. The analysis closest to ours is done by the authors of the ONCE dataset. They analyzed the collected data regarding distance-wise mean Average Precision performance for 3D object detection using only point clouds. However, their distance thresholds were selected rather intuitively, whereas we specifically derive the distance from the domain safety requirements. Additionally, we analyze the spatial distribution of objects within the images as well as the bounding box/object size compared to the image size.
Since the majority of related works only analyze datasets from a general ML perspective, omitting the point of data-centric paradigm, we decided to verify the SOTA automotive datasets in regard to a trajectory planning task. One part of our motivation is that forecasting the trajectory planning is conditioned by ego's vehicle velocity, which in case of higher value takes the further-distance objects into account. In order to be able to evaluate the sufficiency and quality of annotated objects within the datasets, we've chosen the Time-To-Collision as an instance to calculate the minimal safe distance from the ego vehicle.
For the sake of simplicity, we do not consider any obstacle heading from the opposite direction (on the collision course), since we are working with static images and thus don't have the information about the relative motion of the objects. As described in [20]: "The TTC value at instant 𝑡 is defined as the time for two objects to collide if they continue at their present velocity and on the same path". Let's define the first object to be an obstacle (anything else than the ego vehicle) and the second object to be the ego vehicle. We consider the velocity of an obstacle to be equal to 0km/h (representing the stand-still object and therefore the worst-case scenario), and the ego vehicle's deceleration to be 7 𝑚/𝑠 2 (can variate within a range from 7 𝑚/𝑠 2 till 10 𝑚/𝑠 2 on dry roads [21], [22]). Vehicle deceleration can be seen as a function of adhesion between the tires and the road, which depends on the material used in the tires, material of the road, temperature, weather conditions, mounted braking system, and the mass of the vehicle.
Based on the definition of TTC, let us consider three driving scenarios:
• highway (recommended speed 130𝑘𝑚/ℎ ≈ 36𝑚/𝑠)
• country road (maximum speed 100𝑘𝑚/ℎ ≈ 28𝑚/𝑠)
• city (maximum speed 50𝑘𝑚/ℎ ≈ 14𝑚/𝑠)
We now compute the minimal safe distance which needs to be ensured in order to brake in time (without initiating any evasive maneuver), for the following case: ego vehicle is driving on the highway, the possible deceleration is equal to 7𝑚/𝑠 2 and reaction delay is 0.0𝑠. Based on the kinematic equations of a linearly decelerating object, the distance which the object will travel is a function of time 𝑠 = 𝑠0 + 𝑣0𝑡 − 1 2 𝑎𝑡 2 , where time 𝑡 is a function of deceleration 𝑡 = 𝑣 0 −𝑣 𝑎 . With a linear deceleration of 7𝑚/𝑠 2 , the vehicle, moving within the legal limits, will reach its standstill state in time 𝑡 = 𝑣 0 −𝑣 𝑎 = 36−0 7 = 5.14𝑠. Within this time frame, the ego vehicle will travel a distance of 𝑠 = 0 + 36 • 5.14 − 1 2 • 7 • 5.14 2 = 185.04 − 92.46 = 92.58𝑚. For completeness, the braking distance under the same weather conditions on country roads is 55.11𝑚 and in the city 14𝑚 as can be seen in Table 2. As mentioned earlier, this process can be generalized and repeated for It is noticeable that the highway's maximum foresight boundary will be in reality limited by the physical properties of the camera or the maximum speed difference between the ego vehicle and the object. But most importantly, objects within those safe ranges must be part of a dataset (training and testing), otherwise, the system will deal with an epistemic uncertainty [23]. In order to be able to investigate the statistics of objects' appearances, we need a dataset that provides information about the object's distance. As mentioned in Section 2, some datasets such as Perl and BKK100, etc. are not suitable for this task. Consequently, we have chosen the following large-scale automotive datasets: KITTI, Waymo open Dataset, A2D2 from Audi, nuScenes, LyftLevel5 and ONCE, which contain 3D annotations and distances to the objects.
In regard to the functionality of trajectory planning, we focused on the following in-dataset object characteristics:
• distribution of a BB's relative size: in order to verify that a variety of objects' sizes is captured within the dataset,
• distribution of the distance between obstacles and the ego vehicle (with relation to minimum safe distances): in order to verify that objects in further distances are incorporated within the dataset,
• relation between BB's relative size and object distance from ego vehicle: to discover abnormality within the dependency,
• heatmap of an object's appearance density: to visualize the potential asymmetrical appearance of an object with relation to the ego vehicle,
• an optical flow between consecutive images; in order to identify a series of static images, which can lead to class imbalanced dataset.
As originally presented in [19] and further explored in [18], it seems to be reasonable to observe the mean Average Precision with relation to a specific object's size. Since the original authors clustered the object groups rather generally (small, middle, big), we propose to have clearly specified operational domain dependencies and incorporate the minimum safe distances as thresholds. Furthermore, the relation of AP to the objects' relative BB sizes or distance highlights detailed discrepancies between the model's performance trained on the same dataset. We therefore incorporate the minimal safe distance 𝑑𝑠 of each scenario as a threshold for the creation of test subsets 𝒜 ⊂ ℬ ⊂ 𝒞 from original test dataset 𝒟. For instance 𝒜 contains objects only in distance > 𝑑𝑠(ℎ𝑖𝑔ℎ𝑤𝑎𝑦). For each subset the average precision can be calculated, representing a concrete value for a specific operational domain e.g. driving in the city. Equation 1 represents a different perspective on the average precision metric, which we call automotive mean AP.
where 𝑛𝑐 is a number of domain specific test-subsets.
In this chapter we analyze all datasets from Table 1 for the characteristics mentioned in Section 3. In order to evaluate the model's generalization ability, the collected data has to be divided into two parts, namely training and validation. The training part is used for extracting the relevant features and finding a reasonable combination in order to build a high-level feature representation, whereas the validation part is used to evaluate the loss after each training cycle (epoch). The training loop usually ends when the loss, on a validation set, stagnates for several epochs [24]. Logic demands that both the training and validation parts should have uniformly distributed objects densities and their properties (e.g.size, distance). We therefore decided to evaluate the correlation between the theoretical uniform distribution and observed one by calculating the Wasserstein distance [25].
There is no reason to assume that all data are flawlessly annotated since the task is usually done by several people, which increases the uncertainty of an inconsistent annotation style. We therefore designed plausibility check functions which allow us to indicate the potentially wrong annotated data and analyze them later on.
• 𝑓1 : 𝑟𝑒𝑡𝑢𝑟𝑛 (0.0 ≥ 𝑟𝑒𝑙.𝑏𝑏.𝑠𝑖𝑧𝑒 > 1.0)
The function 𝑓1 returns a bounding box whose relative size is out of the range (0.0, 1.0]. The outliers of otherwise exponentially decayed objects' size with respect to the distance to the ego are marked by function 𝑓2. This function covers the quadratic dependency of the BB area and the mapping of the captured object within the real world to the pixel coordinates. The parameter 𝑙𝑜𝑛𝑔.𝑐𝑎𝑚.𝑜𝑓 𝑓 and the denominator 𝑑 were found with the least squares optimization from the analyzed data and therefore are unique for each dataset and each class. The function 𝑓3 highlights objects whose bounding boxes significantly overlap. The threshold of the relative intersection area can be defined by 𝑡ℎ𝑟 𝑜𝑣𝑒𝑟𝑙𝑎𝑝 .
Examples of the outcomes for functions 𝑓1 − 𝑓3 are given in Figure 1. We further found that most of the datasets contain sequences of "stop and go" in a traffic jam, or idling at crossroads. These situations result in the recording of many similar images without objects or surrounding variation. Therefore we added which calculates a dense optical flow [26] from previous and actual images in the sequence and returns a positive flag in case the magnitude sinks bellow empirically defined threshold as seen in Figure 2. The higher the value of the magnitude, the more objects were moving from frame to frame. With this method, we could even identify if the data were recorded repeatedly in the same place [27] (when recorded in one session), but we only used it to discover static scenarios. We further encourage to analyse such a subset of filtered images and adapt the parameters of the outliers function 𝑓 1 − 𝑓 4 to make them suitable to the target domain.
Exemplary results of the analysis of the objects' distances distribution as well as the relation between the relative BB size and object distance can be seen in Figures 3 and 4. Moreover, we show an example of object appearance variation (heatmap) of class Vehicle in the A2D2's dataset in Figure 5. The information from subset (0.0 to 14.0) meters will be used in scenarios where a vehicle is driving less than 50km/h (idling on a crossroads, for instance). However, the A2D2 dataset doesn't contain any objects with the minimal safe distance necessary for highway driving (92.58m and more).
To summarize, most of the SOTA autonomous driving datasets are generated with a focus on a large amount of scenes/driving hours/frames while considering different weather, location, and daylight conditions. However, as we discovered, none of the datasets contains a sufficient amount of information for safe autonomous driving on the highway (described in Section 3). Every analyzed dataset was lacking high-distance annotated objects, as can be seen in Table 3. Such a gap can be explained by the physical limitations of the camera (too low resolution), the annotation style (objects under a certain pixel area were excluded from the annotation process), and the ambient conditions in which the dataset was recorded. In addition, a system trained on such a dataset would have to deal with epistemic uncertainty and look for additional sources of information (namely LIDAR).
Furthermore, all datasets contain predominantly smallsized objects (the highest MEAN value of the relative BB size of the class Person was 0.091 in the case of the KITTI dataset). For comparison, the same can be stated for the well-known COCO dataset [19], where the class Person has a MEAN relative BB size equal to 0.089. By generating heatmaps, we discovered that 99.8% of the objects appear only in the two lower quadrants of the image. Such information can lead to a significant downsizing of the field of view and the thereof acceleration of the detectors' inference time. The majority of overlapping BBs, with potentially wrong annotation styles, were extracted from a sequence of streams on crossroads. Such a static data sequence (9.55% in case of nuScenes dataset) contains a lot of similar features (the majority of surrounding objects are not moving) and could be removed from the dataset.
Finally, we defined and evaluated a reasonable set of rules, described in Section 4.1, which automatically proves the quality of the collected data from a domainrelated perspective. We encourage the community to use our "domain-centric" approach in order to create a dataset under concrete functional constraints and train detectors on it. Our code and additional results are published on GitLab and can be publicly accessed. 1 This work deepened our vision of a domain-centric ML approach in the automotive industry. To conclude, we outline some research directions which we are currently investigating: (a) Analysis of automotive mAP based on 1 https://gitlab.com/arrk-fi/ObjectDetectionCriticality/-/tree/ dependency_branch.
This work is partly funded by ARRK Engineering GmbH. The work has also been supported by the grant of the University of West Bohemia, project No. SGS-2022-017 and by the Technology Agency of the Czech Republic, project No. CK03000179.