Today, in the border and perimeter protection, it is very common to use RADAR technology and pan-tilt (PT) cameras to have terrain dominance. The common solution uses both sources - while most of the threats are detected by radar, the camera used for inspection of motion, detected by radar. This solution is very dependent on radar performance and not efective for diferent scenarios when the radar is not capable to monitor the movement of all targets. Inputs from camera and radar are used in close integration to increase detection probability and reduce false alarms. In this work two alternative methods of radar and visual data fusion are proposed, data structures and processing algorithms are defined and results of experimental validation for both proposed methods are shown.
fusion are defined in this context, with first covering
such important parts of the system like data
associaSensor fusion is a large research topic. Its goal is to tion and state updates and second being more modular
combine multiple data sources to receive joined data, and distributed alternative.
which allows to improve processes or calculations, com- Methods based on Kalman family filters [
Tracking solution using radar as the only source of cause they enable to have process model independent
data sufers from unreliable detection or even absence from observation structure [
This work focuses on research related to the practical application of fusion between radar and video. Two main methods of fusion, namely data fusion and tracks
The first problem can be solved in several ways. The
usual practice is to keep the target state in Cartesian
coordinates [
Second issue is not addressed by these solutions: if
Kalman filter would be used while keeping target state
in Cartesian coordinates, camera would change from
very precise sensor ( azand el angles) to very unprecise
( x, y, z), as distance to object is used in Polar to
Cartesian transform for any direction and is not directly
measured by camera. There are numerous diferent
approaches to get at least some estimation of distance
from direct camera measurements:
1. Use radar detections as a base and map
camera detections to radar improving angular
resolution [
era calibration in the lab.
3. Use corner reflector or another strongly
reflective object to map precisely radar and camera
detections into 3D [
For scenarios, arising in perimeter protection or
homeland security, method 6) is not applicable or too costly
in a sense of performance. Usually, there are no
predeifned markers and the system is stationary (no
translational movement). All other approaches can be
explored. Ideal solution, however, would be to use
camera only in its strongest domain to augment
information received by radar instead of increasing
inaccuracies in one dimension while decreasing in others. One
such potential solution is to keep the state of Kalman
iflter in Polar coordinates. This isn’t unknown in
literature, as bearing only tracking often uses Modified
Polar Coordinates (MPC) [
Following issues are explored before finalizing data model for fusion: 1. The impact of the origin of the coordinates – to use Cartesian or Polar coordinates in the linear Kalman filter, for the diferent movement patterns. 2. Improvement (if any) of the accuracy of state estimation in Polar coordinates, if the camera is also used in Kalman filter updates. 3. Detection of distance to objects from camera measurements and the efect of the addition of this result to the Kalman filter measurement model.
As a consequence, some of the following sections (section II, section V and section VI) are split into two main parts - preliminary experiments (simulation) and methods validation. It should be clear, however, that chronologically all preliminary tests were performed and results were analyzed before finalizing fusion models and implementing fusion methods.
2. Data Description 1A geometrical relation between two images of the same planar Coordinates, used in this research are defined as shown surface, described by the transformation matrix in Fig.1
2The camera is moving by known pattern, the terrain is flat, camera position and angle to the surface are known, etc.
2. Velocity noise standard deviation 3. Detection angle noise standard deviation 4. False detection rate (per simulation area element) 5. Detection view angle 6. Update rate
The following simulation parameters can be defined for the camera: 1. VMD detection box noise (in pixels) 2. Camera resolution 3. Angular field of view 4. VMD false positive rate and area
In this part of the research, the definition of custom Simulation of detections for VMD and radar and subtrajectories for any number of targets as well as radar sequent registration without knowledge of ground truth and video motion detection (VMD) noise models were or noise model is performed. icmanplbeemdeenfinteedd.foTrhraedfaorl:lowing simulation parameters casKeaolmfCaanrtfilteesriasntactoeoirsddienfinaetedsbaynd by[ [ ]T in ]T in Polar coordinates, where 1. and are velocities in x and z dimensions
respectively, 2. and are accelerations 3. - range change rate 4. - azimuth angle 5. and - the rate of change for azimuth and the rate of change of (angle acceleration)
Measurements of radar and camera are simulated using Gaussian noise model. For camera – diferent accuracies of VMD were tested ranging from 1 pixel to 5 pixels.
Measurements are filtered and the mean square error (MSE) is calculated for comparison of estimated positions to actual trajectories.
In Fig.2 test trajectories Trajectory 1 – T1, Trajectory 2 – T2, Trajectory 3 – T3, which were used for evaluation (camera and radar are at the point (0,0) in a − Cartesian coordinate system) are presented.
In case of track fusion inputs for fusion module are 1. list of VMD outputs as bounding boxes [ ℎ and ID of VMD track 2. list of radar tracker outputs as [ of radar track ] and ID
1. list of bounding boxes [ ℎ ] received directly from the "blob" stage of VMD pipeline 2. list of radar targets [ ]
The main diference between sources of data for two approaches is, that in case of data fusion there are many false detections, which are not filtered by VMD or radar tracker methods respectively.
2. decentralized architecture, 3. distributed architecture, 4. hierarchical architecture.
Centralized architecture (data from all sources is processed in single module) is expected to be theoretically optimal in case of proper synchronization of data sources and suficient bandwidth for data transfer. It can sufer, however, from lack of distribution of bandwidth and processing in case these resources are limited for given task. Alternatives, solving this issue, ] raaretedreacwendtraatalizineddaiferrcehnitteoctrudreer (afnudsioconmnpoodseitsioinnc)oarnpdodistributed architecture (fusion nodes receive single sensor data and provide features to be fused). In our view, decentralized architecture would introduce unnecessary complexity and implementation dificulty, so distributed architecture is considered as only another option to centralized architecture.
Based on the discussion in the previous section and the results of preliminary evaluations (described in section VI), two main fusion approaches are established:
2. tracks fusion
Overview of diferent sensor fusion classifications can Data fusion is a centralized mixed input (DAI-FEO
be found in [
Possible schemes of the sensor fusion in a given ap- detected blobs). plication are limited to redundant schemes (with other Tracks fusion is distributed FEI-FEO method, acceptpossibilities being complementary and cooperative) bas- ing features (tracks) from VMD and radar tracker moded on the relations of sensors used in the system. Both ules. types of sensors are measuring the same state of ob- Both should solve the problem defined, but the pros jects at the same time in the observed area. and cons of approaches are diferent. Fusion of tracks
Based on the idea of modularity of the system, it is is performed after data from both sources is processed
clear, that radar/camera fusion should not be a decision- and there is track detected on both sets of data. Then
making module. While detections are performed by both tracks are matched (track to track association) to
the fusion module, the final decision is afected by ad- better reflect the behavior of the object being tracked.
ditional rule-based filters and the recognition module. In the case of one of the sources not returning track
The goal of the fusion module is to minimize the false while other is returning track, diferent policies can
alarm rate (FAR) of the system and provide inputs for be used to favor reduced false detection rate over
exdecision-making modules. It can be said then, that fu- tended tracking duration or vice versa. Comparing
sion module approaches can be limited to two (data in with data fusion, tracks fusion is easier to debug and
- feature out (DAI-FEO), f eature in - feature out (FEI- tune, since separate modules can be tested and tuned
FEO)), contrary to approaches, which provide data or faster due to overall reduced complexity. Data fusion
decisions as outputs (data in - data out (DAI-DAO), works on a lower level than tracks fusion. It has the
f eature in - decision out (FEI-DEO), decision in - de- benefit of incorporating updates from both sources of
cision out (DEI-DEO)). data into common state update converging to a true
Four types of fusion architectures are defined in [
There is no need to tune policies for diferent cases (no recent radar detection, no recent camera detection, higher than an average mismatch between sources, etc.), it is handled by common update. The downside of data fusion compared to track fusion is longer debugging and tuning. If conditions of experiments/use cases or equipment changes, that could add delivery time overhead.
General strategy for data-level fusion can be summarized as follows: 1. Use radar detections as a base for track valida
tion. 2. Any VMD detection can create a persistent track, but it will be validated without radar data only after a relatively long track age is reached. 3. Any track with both recent VMD and Radar detections has a higher probability to be validated as a real track, as a consequence, it is validated at the lower age of the track.
Such choices are proposed due to relative ease of radar data validation - if the movement of a potential target in the area under test contradicts Doppler velocity, reported by radar, such target can be quickly invalidated. There is no similar process for VMD.
Full track state representation consists of the following parts: 1. Track state vector at time (based on definitions in Section II). 6x1 vector: ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ , ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎡ ⎤ ⎢ ⎥⎥⎥ . = ⎢ ⎢ ⎣ ⎦ or its counterpart in Polar coordinates. In case of constant velocity model (4x1 vector): (1) (2) 2. The track covariance matrix . 6x6 matrix or 4x4 matrix (if accelerations removed), describing the amount of variance in data and interstate dependencies (covariance). 3. The track age. Time elapsed from the moment of track creation by VMD or radar. It is updated if either VMD or radar detection can be associated with the current state vector. 4. The track innovation error. Value, which is compared to predefined threshold values for track validation and removal. Track innovation error (squared Mahalanobis distance) is calculated from state measurement residual (innovation) and residual (innovation) covariance. It is one of the main criteria for positive detection. 5. Track update timestamp for VMD 6. Track update timestamp for radar 7. Object size. It can be calculated if both VMD and
radar detected an object. 8. Object visual distance state. 2x1 vector (distance, rate of distance). It is estimated from the camera and can be used if the positioning of the device is known. It is highly unstable due to partial obstructions and because of that is separated from the track state vector. It can be used by higherlevel decision-making modules. 9. Object visual distance covariance matrix. 2x2
matrix. 10. The total duration of detection for VMD. It is one
of the main criteria for positive detection. 11. The total duration of detection for radar. It is
one of the main criteria for positive detection.
The fusion scheme consists of data association, state update, and management of tracks. Data association (point to track) in the first prototype is implemented as simple Nearest Neighbour (NN) estimation in state space, favoring simplicity and speed. Joint Probability Data Association (JPDA) based association method [24] is used as an alternative in later versions. In NN based data association each measurement is compared against the estimated state | −1 at time by calculating Mahalanobis distance based metric (discussed further). The prefiltering of potential associations can be performed using some simple heuristics like 2D distance.
Linear KF is used for state estimation and update.
Estimation step: | −1 = −1| −1,
(3) where A is process matrix defined for a state with accelerations as: for state without acceleration as: = ⎢ ⎡1 ⎢0 ⎢ ⎢0 0 0 0 ⎢ ⎢ ⎣ 1 0 0 0 0 1 0 0 0 2 = ⎢ ⎡1 ⎢0 0 ⎢ ⎣0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0
measurement matrix, is the measurement vector of x1 size. The size of the measurement vector and the measurement matrix depend on the type of sensor and coordinate systems used for measurements and the state. can be understood as number of parameters measured by a specific sensor and as mapping between state and measured parameters.
If transformation from sensor coordinates to state coordinates is linear, it can be performed by In our experiment of using Polar representation of tar
directly. get state space = ⎢ ⎡ ⎤ ⎢ ⎥ ⎢ ⎣ ⎥ . ⎥ ⎦ = ⎢ ⎥ , ⎡ ⎤ ⎢ ⎣ ⎦
⎥ The data vector of radar and measurement matrix: which simplifies calculations. In the case of camera (VMD) update without the knowledge of the geometrical setup:
= [ ] . here then
is just x position of the center of the bounding box in screen coordinates. The measurement matrix = [0 0
H FOV 0 also be calculated by just using and the state vector needs to be augmented with additional element 1 as the last row to allow matrix multiplication in (7). In the above formula, is the width of the video in pixels. The single element matrix
can element of the state ⎢ ⎣ matching pair if all are larger than some predefined
identity matrix. The best match be(4) (5) (7) (8) (9) here is x diagonal measurement error matrix. It is defined based on the parameters of sensors used. Error ranges from the datasheet of the sensor is a good starting point. Kalman gain
= | −1 − T −1, here inverse of is taken. Innovation error, which is not used directly in the Kalman filter update, but is the main criterion for data association:
= T −1 ,
Lastly, if it is found, that there is the best match between the track-measurement pair, the Kalman filter update step is finished by calculating the posterior state and covariance: threshold . In such a case, a new track state for the track without measurement: (19) (20) (21) | = | −1, | = | −1.
Track merging is performed if (similarly to (16)):
= ( 1 − 2 )T m−1atch( 1 − 2 )) < thresh, where 1 and 2 are states of two tracks to be compared, match is a diagonal matrix, can be treated as typical variances of track state elements (with large values of match more tracks will be matched), thresh is a threshold for matching.
With the main parts of the data association and track update discussed, global view of data fusion can be deifned (see Algorithm 1).
During the step of managing unmatched tracks, innovation error of track (normally calculated as (16)), is increased. In current implementation following empirically found formula is used:
√ = ∗ max(1 + /3, 1 + 6 / tr),
(22) here tr is the age of the track. The idea is to increase for new tracks faster, that older tracks, as the track age usually shows, how reliable the current track is.
The age of track is increased if there is VMD or Radar detection. If detection was from previous measurement (contrary to measurements, which came from both sources on the same processing step) it is increased by the time diference between current and previous measurements. If track is picked up after an absence of matching measurements, delta time is added based on the type of update (frame update time for VMD or frame update time for Radar). Tracks without current matching measurements do not update age value.
For VMD only state (no recent radar detection) previous angular movement is used along with size to create view space gating for measurement to tracks matching. It is part of measurement prefiltering, mentioned earlier. If angular velocity isn’t initialized (the track age is low), the maximum possible rate of movement based on typical size and velocity ratio is used to create view space gate.
For a track, having recent Radar detection or overall high radar detection duration, the exact estimated position is calculated and new detections of VMD and Radar are projected on common space to use spatial gating.
Tracks are created/initialized with every moving object detection. For radar detection movement condition is non-zero Doppler velocity by default or can /* merging of existing tracks for each track t in memory do for each track t2 in memory except t do apply (21); if (21) condition met then merge t and t2; remove t2 from memory;
*/ end end end /* updating of tracks by
measurement */ if have a new measurement of any source then for each track t in memory do estimate prior state (3) and covariance (6) prefilter list of possible measurements; for each measurement with eps small enough do if measurement was already used then check for more precise updates in earlier tracks; if no better found then update Kalman state (17), (18); push updated state to stack of potential updates; mark measurement as used; end else end end update Kalman state (17), (18); push updated state to stack of potential updates; mark measurement as used; end for each track t in memory do apply the best update from the updated states stack; end end end /* manage unmatched tracks for each unmatched track do increase track innovation error; if innovation error reaches threshold then
remove a track from memory end /* create potential tracks for each unmatched measurement do if satisfies movement conditions then create a track with an initial state and initial covariance; */ */ 98
end end
Algorithm 1: Data fusion algorithm be set as one of the many algorithm parameters. All VMD detections treated as moving by definition. After the creation track is in a non-validated state. Tracks are considered to be validated after predefined age depending on the following properties: 1. Calculate angle to the base of target based on the bottom edge of VMD detection and knowledge of VFOV of the camera as: = ℎVFOV
, ′ = − .
= tan ′. where ℎ is projection of position, where target touches the ground on camera view, - vertical resolution (number of pixels along the vertical axis of image). 2. Subtract this angle from the angle formed by straight
up direction and camera "looking" direction: 3. Calculate distance, by knowing one side of the triangle (height of camera) and the angle between this side and hypotenuse: (23) (24) (25) 1. Existence or absence of recent VMD/Radar detection. Radar detections (without VMD detec- There are other ways to estimate the distance to the tion) having a higher impact on validation thresh- target from the camera only. For example, if the tarold than another way around. get is identified, the relative size of the target on the 2. Track innovation error. image could signal distance. That requires, however, 3. Total previous duration of detections for radar a feedback loop between the fusion module and the and VMD recognition module. 4. Trajectory type - mostly tangential movement with radar only detections should be verified for 4.3. Tracks Fusion a longer duration. 5. Velocity thresholds.
A general strategy for tracks fusion can be summarized as follows: Tracks to be removed from potential tracks list if: 1. Track innovation error grows too large. It is calculated based on new measurements and also incremented after the absence of radar detection (22). 2. The track is created by VMD detection and visualonly innovation error grows to large. It is calculated from angular measurements only and updated similarly to (22).
Distance from camera measurements is calculated based on assumptions, that: 1. target is not blocked by some other objects 2. height and elevation of the camera are known 3. detection is of ground-based targets
The main features used for calculations are presented in Fig.3. Algorithm: 1. Both sources can create fusion tracks with another component not present until the match will be found at a later stage. 2. Input tracks are not verified against Kalman state estimate (no data association), because this step is already done in the tracker module. 3. VMD and radar tracker tracks can be merged if merging requirements are met. From such moment fusion track has both components. 4. The track can be split, if it is detected, that visual and radar data diverge too much. Track splitting/merging is performed at each data update step.
Additionally to track structure discussed in the previous section following fields are defined for tracking state: 1. Fused track ID (diferent from components). 2. visualSeparated- boolean value, which shows, that track recently had a visual component but lost it due to diverging of visual and radar. 3. VMD track ID. With new data updates, the degree of matching between fused tracks components is first checked for stored best previous matches 4. Radar tracker track ID. Same as above.
The track fusion algorithm is overviewed in Algorithm 2. The operation of mismatch calculation, used in algorithm description on many occasions can be explained looking at Fig. 4. First, measurement timestamps are created for all entries of both types of tracks. Then estimations for matching are calculated by interpolation (or extrapolation, if on edges). Average azimuth mismatch is used as a matching parameter. The early exit of the matching function is possible if mismatch grows to a predefined value.
The age of track is updated as per data fusion with each new radar tracker output considered as new radar measurement with time step equal to radar update duration. Track time out is increased, if no measurements were added to track. This step is the same for component tracks and the fused track. Time out for deletion calculation, mentioned in Algorithm 2, is calculated based on the current number of tracks. It is deifned as 3 s if the number of tracks is less than max the maximum number of tracks. If, on the other hand, the number of tracks is higher, allowed time out reduces: timeout = 3 2 max − cur max .
(26) This assures, that all tracks are cleared if cur reaches 2 max. If a number of tracks for some reason grows more than 2 max, all tracks are cleared.
Tracks are created/initialized with every moving object detection from radar and every VMD detection. After creation, VMD track is in non-validated state, but track, created from radar tracker data directly, is in a validated state. Fused track having both components can be split, if VMD measurements diverge from tracker output too much. visualSeparated is set to true in this case. It is done to prevent the reacquisition of the same VMD track with a high mismatch factor. VMD only part of such split inherits range data and all */ */ */ */ /* updating tracks structures if have a new radar tracker frame then for each track in frame do if The ID of a track can be found in already existing then
append a new measurement; else end end create a new list of track entries with new ID; end if have new frame of VMD tracks then
same as for radar tracks; end /* matching of tracks for each fused track do calculate mismatch of radar and video; for all non-fused tracks of both types do calculate mismatch with appropriate (radar vs. VMD) track; if a better match found then assign a new component to fused; release the previous component as non-fused; end end end /* generation of tracks for each combination of radar and VMD track do */ calculate mismatch; if mismatch small enough then create a new fused track with matched components end end /* destruction of tracks calculate time out of track for deletion; delete all tracks with higher time out than allowed; /* tracks state update for each fused track do if any of the track components received updates then
a full Kalman filter update else end end update the state as an estimation only (17),(18) Algorithm 2: The track fusion algorithm data, which is relevant to durations of detections and age of the track. It can be matched again later after visualSeparated expires. The split tracks get invalidated for some short duration (less than second) by setting it’s innovation error parameter to some high value and gradually reducing it after new measurements. data acquisition was performed simultaneously. Time synchronization was assured by knowing the starting time of video and frame rate and storing radar raw data or radar tracks with exact timestamps. Although the discussed algorithms were not running at the time of these measurements, close to real-time performance is achieved later using the same hardware.
allows more control over experiments.
Three experiments are performed: The area, selected for experiments is shown in Fig. 5.
The selected area allows the testing performance of tracking on big enough distance (more than 100 m) with parts of trajectories being almost exactly tangen- Experimental investigation was subdivided into two tial while moving around the edge of the stadium. A stages. First, preliminary experiments were performed small amount of moving background (cars, people, trees) to select the state model and update strategy for fused tracks. During the second stage, the tracks fusion and the data fusion approaches were validated.
1. A person moving clockwise around the edge of the stadium without stopping 2. A person moving counter-clockwise around the edge of the stadium, stopping, then proceeding 3. A person moving clockwise around the edge of the stadium, then changing the moving pattern from mostly tangential to radial
An example frame of video with detection displayed is presented in Fig. 6. Data were acquired using the NXP iMX6 SoM based embedded system with a quadcore 1.2 GHz processor. Video recording and radar
The impact of the selected system of coordinates to performance is presented in (a) - (c) pictures in Fig. 7. KF state representation by Cartesian or Polar coordinates produces very close results and a visual separation of results is hardly noticeable. In Table 1, the results of models, in which Polar or Cartesian coordinates are used for KF state representation performances, are presented. Since results are very similar, it can be concluded, that there is no significant diference in diferent KF state representations. To obtain error for each model, 10 simulations were performed for each and mean MSE calculated.
The resulting performances of models with/without adding camera to state update, represented through MSE, are shown in Table 2. It can be observed, that KF state updated using camera output represents ground truth more accurately than updated by radar data only. As before, mean MSE is calculated by running simulation 10 times for each trajectory.
Trajectory
T1 T2 T3
Measured error KF error Polar Fusion error 2.4647 0.87358 0.72844 2.2787 0.58846 0.51873 2.7926 0.63428 0.58261
Results of fusion error and fusion with distance models performance evaluation using MSE statistics are presented in Table 3. The best minimum values as well as mean values of MSE shows, that fusion with distance evaluation model performance outperforms model without distance evaluation performance.
An example of a typical simulation run with diferent models evaluated is shown in Fig. 8
The main metrics for evaluation of two fusion approaches were Object count accuracy (OCA) and FAR. OCA is Trajectory T1 T2 T3
Measured error
KF error
Fusion error Fusion with DIST error Fusion with DIST error
Measured error
KF error
Fusion error Measured error
KF error
Fusion error Fusion with DIST error
MIN Fusion error and fusion with distance estimation error comparison using MSE defined as
OCA ( , ) = min( , ) + 2 , (27) and
where and are sets of ground truth points and detected points in measurement frame respectively,
are quantities of ground truth and detected instances respectively. Overall OCA is defined as the average OCA of all frames of measurements.
MAX
Any false track appearing in the frame constitutes to given frame becoming a false positive.
The focus of the experimental investigation was on elimination of false detections and reduction of missed detections rate. Progress towards both goals can be successfully monitored using selected metrics [25, 26].
Rather conservative policies for tracks validation were chosen for both versions of fusion to highlight the possibility to avoid false alarms while still keeping high enough detection rate (indirectly shown by OCA) for all practical purposes. Evaluation results are presented in Table 4. Best results are obtained by Data fusion and close to the best results are obtained by Tracks fusion. 7. Conclusions 1. It was observed experimentally, that radar only tracking sufers from many missed detections, if the target trajectory is close to tangential. 2. While radar only tracker performs without false alarms during the first two sequences, it is demonstrated with the third sequence, that target direction changes can cause false tracks to appear. 3. Two issues, mentioned above, can be solved with any of two fusion of radar and camera approaches, as it is seen from evaluation results. OCA increased drastically in both cases compared to radar only tracking. 4. Data fusion ofers slightly better performance, reflected by higher OCA values. In practice, it means faster track validation and more robust tracking with missed detections from either VMD or radar. 5. The addition of distance measurements from the camera didn’t prove to be stable method for tracks matching or state updates in practice. Although simulation was suggesting accuracy improvement, real measurements were highly unstable while using this approach.
Research is partially funded by Lithuanian Research Council Project Nr. 09.3.3-ESFA-V-711-01-0001, and partially funded by Lithuanian Business Support Agency Project Nr. 01.2.1-LVPA-T-848-01-0002.