This article proposes a novel concept to leverage the timeconsuming labeling process for training object detectors in automated driving. The approach uses pre-trained probabilistic, well-calibrated object detectors for different sensor modalities. Based on the knowledge about the sensor extrinsics, the probabilistic detections are transformed from one sensor modality into another. These transformed detections are then used as pre-labels for the respective sensor modality. However, these pre-labels are error-prone, such that we propose an additional dedicated labeling quality assessment. The latter allows us to attach a quality seal to automatically pre-labeled data sets and is the starting point for interactive human-in-the-loop learning.
Artificial intelligence and, in particular, machine learning (ML) are the enabling
technologies in autonomous driving. In this context, ML and deep learning
techniques are already successfully used for perception, i.e., sensory environment
and object recognition [
Contributions: We address this problem by proposing a detailed concept to automatically generate pre-labelings via cross-domain label transfer for perception in autonomous driving. Therefore, we identify four major research questions arising within our concept’s stages and provide ideas for targeting each of them. We envision our concept as an application-driven starting point for human-inthe-loop learning. In this context, our concept provides methods leveraging interactive learning techniques in object detection, e.g., probabilistic object detectors, improving the labeling quality, coping with imperfect labels, and decreasing annotation effort. As another major contribution, we see the quality assessment of pre-labelings to support subsequent human-in-the-loop learning processes. In this sense, we aim to provide a quality seal to pre-labeled data sets, e.g., prelabels having an expected mean average precision of 90%. 2
This section describes the four stages of our concept (cf. Fig. 1) and formulates research questions. In the first stage, we develop probabilistic object detectors. The second stage aims to improve the calibration of these detectors further. Subsequently, the probabilistic predictions are interpreted as pre-labels of the different sensor modalities and optionally fused in the third stage. Finally, the concept is concluded by the fourth stage, including a labeling quality assessment based on probabilistic outputs. It serves as a starting point for human-in-the-loop learning to refine the pre-labels and release them for further model training.
Camera LiDAR
Images
Stage I Probabilistic Object Detector Probabilistic Object Detector
Probabilistic Class Labels
+ Probabilistic 3D Bounding
Boxes New Data for Training
Stage II Probability Calibration Probability Calibration
Labeled Data
Calibrated Class Labels
+ Calibrated 3D Bounding
Boxes
Stage III Probabilistic Sensor Fusion
Label
Transfer Annotator
Human in the Loop
Pre-labeled Data
Stage IV Labeling Quality
Assessment Fig. 1. Illustration of the proposed highly automated labeling process exemplary shown for two sensors. Dashed arrows represent optional processes.
A Concept fHorigHhilgyhAlyuAtoumtoamteadtePdrPe-rLea-LbaeblinelginvgiaviCarCorsoss-Ds-oDmomainainLaLbaebleTlTrarnasnfsefrer
Stage I: Probabilistic Object Detection – How to train probabilistic object
detectors for different sensor modalities? In stage I, we consider pre-trained
object detectors for different sensor domains, i.e., camera- and LiDAR-based object
detectors [
Stage III: Label-Transfer and Probabilistic Sensor Fusion – How to
transfer labels between different sensor modalities and combine probabilistic
predictions originating from different sensor modalities? In stage III, we use the
extrinsic sensor parameters to transfer pre-labels from one sensor domain to
another. Therefore, we assume that the sensor extrinsics are known in advance. The
transferred pre-labels can be used as labels for the other sensor modality and
vice-versa. Moreover, we also investigate the fusion of probabilistic pre-labels
(i.e., detections) originating from different sensor modalities (cf. stage I and II).
The fusion is realized employing a Bayesian approach (cf. [
This article presents a novel concept for highly automated pre-labeling via crossdomain label transfer for perception in autonomous driving. The novelty of our concept lies in the label transfer exploiting the strengths and weaknesses of different sensor modalities for object detection. The use of multiple sensors to improve perception is not new. However, the use for pre-labeling (in the context of 3D bounding box detection) in combination with an explicit quality assessment component under consideration of calibrated probabilistic predictions represents a novel approach. It allows us to attach a quality seal to pre-labeled data sets. The quality assessment is the starting point for human-in-the-loop learning and iterative model improvement. Although our concept focuses on the autonomous driving domain with LiDAR and camera sensors, it can be extended toward multiple sensors and possibly different applications involving data from multiple sensors. Moreover, the presented ideas form a foundation for further investigations in the area of interactive adaptive learning. For example, the uncertainty estimates of the developed probabilistic object detectors might be used to derive novel utility measures for active learning in object detection.
This work results from the project KI Data Tooling (19A20001O) funded by German Federal Ministry for Economic Affairs and Energy (BMWI) and the DeCoInt2-project financed by the German Research Foundation (DFG) within the priority program SPP 1835: “Kooperativ interagierende Automobile”, grant number SI 674/11-2. We, in particular, thank Kristina Dingel, Alice MoallemyOureh, and Daniel Kottke for their insightful feedback.