Real-time tracking of multiple objects with locally adaptive correlation filters A.N. Ruchay V.I. Kober I.E. Chernoskulov 2017 214 218

A tracking algorithm using locally adaptive correlation filtering is proposed. The algorithm is designed to track multiple objects with invariance to pose, occlusion, clutter, and illumination variations. The algorithm employs a prediction scheme and composite correlation filters. The filters are synthesized with the help of an iterative algorithm, which optimizes discrimination capability for each target. The filters are adapted online to targets changes using information of current and past scene frames. Results obtained with the proposed algorithm using real-life scenes, are presented and compared with those obtained with state-of-the-art tracking methods in terms of detection efficiency, tracking accuracy, and speed of processing.

 tracking locally adaptive filters correlation filters matching 1. Introduction 2. Composite filter design using optimum correlation filters

We are interested in the design of a correlation filter which is able to recognize an object embedded into a disjoint background in the scene corrupted with additive noise. The designed filter should be also able to recognize geometrically distorted versions of the target. Let T = {ti(x, y); i = 1, . . . , N} be an image set containing geometrically distorted versions of the target to be recognized. The input scene is assumed to be composed by the target t(x, y) embedded into a disjoint background b(x, y) at unknown coordinates (τx, τy), and the whole scene is corrupted with additive noise n(x, y), as follows:

H∗(u, v) =

T (u, v) + μbW(u, v)

Pb(u, v) ⊗ |W(u, v)|2 + Pn(u, v). Let us denote by

p = Ra.  T   u = 1, . . . , 1, 0, . . . , 0 , |N{onzes} |M{zezros} u = Q+p, u = Q+Ra. a = [Q+R]−1u. p = R[Q+R]−1u.

|cb|2

DC = 1 − |ct|2 , the desired responses to the training patterns, and denote by Q the matrix whose columns are the elements of U. The response constraints can be expressed as where superscript + denotes complex conjugate. Substituting ( 4 ) into ( 5 ), we obtain

Thus, the solution for a, is Finally, substituting ( 8 ) into ( 4 ), the solution for the composite filter is given by

In ( 2 ), T (u, v) and W(u, v) are the Fourier transforms of t(x, y) and w(x, y), respectively; μb is the mean value of the background b(x, y); Pb(u, v) and Pn(u, v) denote power spectral densities of b0(x, y) = b(x, y) − μb and n(x, y), respectively. The symbol ⊗ denotes convolution.

Let hi(x, y) be the impulse response of a GMF constructed for the ith available view of the target ti(x, y) in T . Let H = {hi(x, y); i = 1, . . . , N} be the set of all GMF impulse responses constructed for all training images ti(x, y). Additionally, let S = {si(x, y); i = 1, . . . , M} be an image set containing M unwanted patterns to be rejected. We want to synthesize a filter capable to recognize all target views in T and to reject the false patterns in S , by combining the optimum filter templates contained in H, and by using only a single correlation operation. The required filter p(x, y), can be constructed as follows [26]:

N N+M p(x, y) = ∑ αihi(x, y) + ∑ αi si(x, y),

i=1 i=N+1 where the coefficients {αi; i = 1, . . . , N + M} are chosen to satisfy prespecified output values for each pattern in U = T ∪ S . Using vectormatrix notation, we denote by R a matrix with N + M columns, where each column is the vector version of each element of U. Let a = [αi; i = 1, . . . , N + M]T be a vector of coefficients. Thus, ( 3 ) can be rewritten as

Note that the value of the correlation peak when using the filter given in Eq. 7, is expected to be close to unity for true-class objects, and close to zero for false-class objects.

The discrimination capability (DC) is a measure of the ability of the filter to distinguish a target from unwanted objects; it is defined by [26] where cb is the value of the maximum correlation sidelobe in background area and ct is the value of the correlation peak generated by the target. A DC value close to unity indicates that the filter has a good capability to distinguish between the target and any false object. Negatives values of the DC indicate that the filter is unable to detect the target. Also, if the obtained DC is greater than a prespecified threshold (DC > DCth), then the target is considered as detected and, otherwise, the target is rejected.

 3. Object tracking with locally adaptive correlation filtering

In this section we describe the proposed algorithm for object tracking based on composite correlation filtering. The proposed algorithm is robust to pose changes and appearance modifications of objects, as well as to the presence of scene noise, illumination changes, and target occlusions.

The algorithm starts with an initialization step where the objects are selected. Next, an optimum correlation filter for reliable detection and location estimation of the target is designed. Afterwards, a composite locally adaptive correlation filter is synthesized. The proposed algorithm incorporates an automatic re-initialization mechanism that reestablishes the tracking if it fails. The block diagram of the proposed algorithm is depicted in Fig. 1. The detailed operation steps are explained below. Step 1: For each object select a small target ti(x, y) from a captured scene frame fi(x, y) containing the object to be tracked. Step 2: Synthesize an optimum correlation filter hi(x, y) with ( 2 ) for reliable detection and location estimation of the target ti(x, y) in the observed local frame li(x, y). ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) Step 3: Synthesize a composite locally adaptive correlation filter pi(x, y) as follows. First, detect and locate the target by hi(x, y) filter from the observed local frame li(x, y). If the obtained DC is greater than a prespecified threshold (DC > DCrec), then the target is considered as successfully detected, ti(x, y) added into the set T and recursion should be stopped. Otherwise, the target si(x, y) corresponding to a false peak added into the set S . Second, synthesize a composite filter pi(x, y) with the help of ( 7 ). Third, detect and locate the target by pi(x, y) filter from the observed local frame li(x, y) recursively until the condition DC > DCrec is satisfied.

Step 4: Detect and locate the target in the observed local frame li+1(x, y) from a new scene frame fi+1(x, y) by pi(x, y) filter. The coordinates of the observed local frame li+1(x, y) are provided by a prediction process that analyzes the motion kinematics of the target. If the obtained DC is greater than a prespecified threshold (DC > DCth), then the target is considered as successfully detected and pi(x, y) filter added to the bank B of composite correlation filters. Otherwise, the target is lost in the observed local frame li+1(x, y) and we recursively used the filters from bank B until condition DC > DCcon is satisfied. The filter from bank B with condition DC > DCcon is used to a new scene frame. If the target is lost in the observed local frame li+1(x, y) with help the filters from bank B, then the coordinates of the target is set coordinates of the past scene frame fi(x, y) and we proceed to a new scene frame fi+2(x, y).

Begin Capture a scene frame fi(x,y) Select the target ti(x,y) Synthesize an optimum correlation ✁lter hi(x, y) Add different versions of target Locate next local frame li+1(x, y) by motion kinematics in fi(x,y) Yes Compute DC with li+1(x,y) Select local frame li(x,y) Construct composite filter pi(x,y) by SDF Compute DC with frame fi(x,y) DC>DCrec DC > DCth Yes Add to the bank of composite correlation filters Proceed to new frame fi+2(x, y) No No

Yes Add new rejection pattern

 Create a new rejection pattern from background Locate maximum in correlation plane Select previous filter from the bank of composite correlation filters Compute DC on li+1(x,y) DC > DCcon No 4. Computer simulation

In this section, computer simulation results obtained with the proposed algorithm for object tracking are presented and compared with common algorithms in terms of detection efficiency, tracking accuracy, and speed of processing.

In order to evaluate the performance of our tracker, we conduct experiments on 100 challenging image sequences from Object Tracking Benchmark (TB-100 database) [27]. These sequences cover most challenging situations in object tracking: Illumination Variation (IV), Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Motion Blur (MB), Fast Motion (FM), In-Plane Rotation (IPR), Out-of-Plane Rotation (OPR), Out-of-View (OV), Background Clutters (BC), Low Resolution (LR).

For comparison, we run 3 state-of-the-art algorithms with the same initial position of the target. The first tracking algorithm (SURF) [28] is based on matching of local features and descriptors. The second tracking algorithm (STRUCK) predicts the target location change between frames on the basis of structured learning [29]. The third collaborative tracking algorithm (SCM) is combined a sparsity-based discriminative classifier and a sparsity-based generative model [30]. The work [27] performed largescale experiments to evaluate the performance of recent 33 object-tracking algorithms. Tracking algorithms STRUCK and SCM perform much better than the others.

For evaluating of detection efficiency we use an evaluation metric of the overlap score. Given a tracked bounding box rt and the ground-truth bounding extent r0 of a target object, the overlap score is defined as

S = ∥rt ∩ r0∥ ∥rt ∪ r0∥ , where ∩ and ∪ represent the intersection and union operators, respectively, and ∥ · ∥ denotes the number of pixels in a region. This average overlap score (AOS) can be used as the performance measure. In addition, the overlap scores can be used for determining whether an algorithm successfully tracks a target in a frame, by testing whether S is larger than a threshold of 0.5. Also we evaluate the tracking algorithms using the average center location error (ACLE) for all image sequences from database.

Table 1 shows the average overlap score (AOS), the average center location errors (ACLE) and the Average Processing Time (APT) on a scena for all the tracking algorithms with the overlap threshold of 0.5. The evaluation results show that our proposed algorithm is faster than the others and more accurate in terms of the average center location errors. When an object moves fastly on the FM subset, the proposed algorithm performs much better than the others. However, the proposed algorithm does not perform well in the subset (IV, OCC, OV) due to illumination variation, and partial occlusion of the target. On the other subsets, the Struck, SCM, and the proposed algorithms outperform other the state-of-the-art algorithms. Fig. 2 shows sample tracking results of the proposed algorithms where the target objects are marked with red rectangles and the actually tracked objects by the proposed algorithm are marked with green rectangles. ( 8 )

5. Conclusion Acknowledgments

A tracking algorithm using locally adaptive correlation filtering is proposed. The algorithm is designed to track multiple objects with invariance to pose, partial occlusion, clutter, and illumination variations. The algorithm employs a prediction scheme and composite correlation filters. The filters are synthesized with the help of an iterative algorithm, which optimizes discrimination capability for each target. The filters are adapted online to targets changes using information of current and past scene frames. The evaluation results show that our proposed algorithm is faster than the others and more accurate in terms of the average center location errors. On the majority test sets the proposed algorithm performs much better than the state-of-the-art algorithms.

This work was supported by the Russian Science Foundation, grant no. 15-19-10010.

 [1] Karasulu , B. Performance Evaluation Software: Moving Object Detection and Tracking in Videos [Text] / B. Karasulu , S. Korukoglu. ”- New York : Springer, 2013 . [2] Talmale , S. Object tracking in images and videos [Text] / S.K. Talmale, N.J. Janwe // International Journal Of Engineering And Computer Science. ”- 2016 . ”- Vol. 5 ( 1 ). ”- P. 15482 - 15486 . [3] Accurate three -dimensional pose recognition from monocular images using template matched filtering [Text] / Kenia Picos, Victor H. Diaz-Ramirez , Vitaly Kober [et al.] // Optical Engineering. ”- 2016 . ”- Vol. 55 , no. 6. ”- P. 063102 . [4] Echeagaray-Patron , B. A. Conformal parameterization and curvature analysis for 3d facial recognition [Text] / B. A. Echeagaray-Patron , D. MiramontesJaramillo , V. Kober // 2015 International Conference on Computational Science and Computational Intelligence (CSCI). ”- [S. l. : s. n.], 2015 . ”- P. 843 - 844 . [5] Echeagaray-Patron , B. A. 3d face recognition based on matching of facial surfaces [Text] / Beatriz A . Echeagaray-Patron , Vitaly Kober. ”- Vol. 9598 . ”- [S. l. : s. n.], 2015 . ”- P. 95980V -95980V-8. [6] Diaz-Escobar , J. A robust hog-based descriptor for pattern recognition [Text] / Julia Diaz-Escobar, Vitaly Kober. ”- Vol. 9971 . ”- [S. l. : s. n.], 2016 . ”- P. 99712A -99712A-7. [7] Diaz-Escobar , J. Text Detection in Digital Images Captured with Low Resolution Under Nonuniform Illumination Conditions [Text] / Julia Diaz-Escobar, Vitaly Kober // Pattern Recognition: 8th Mexican Conference, MCPR 2016 , Guanajuato , Mexico, June 22-25, 2016 . Proceedings / Ed. by Jose´ Francisco Mart´ ınez -Trinidad, Jes u´s Ariel Carrasco-Ochoa, Victor Ayala Ramirez [et al.]. ”- Cham : Springer International Publishing, 2016 . ”- P. 3 - 12 . [8] An efficient algorithm for matching of slam video sequences [Text] / Jose A. Gonzalez-Fraga , Victor H. Diaz-Ramirez , Vitaly Kober [et al.]. ”- Vol. 9971 . ”- [S. l. : s. n.], 2016 . ”- P. 99712Z -99712Z-10. [9] Effective indexing for face recognition [Text] / I. Sochenkov, A. Sochenkova , A. Vokhmintsev [et al.]. ”- Vol. 9971 . ”- [S. l. : s. n.], 2016 . ”- P. 997124 - 997124-9. [10] Face recognition based on a matching algorithm with recursive calculation of oriented gradient histograms [Text] / A. V. Vokhmintcev , I. V. Sochenkov , V. V. Kuznetsov , D. V. Tikhonkikh // Doklady Mathematics. ”- 2016 . ”- Vol. 93 , no. 1. ”- P. 37 - 41 . [11] Tihonkih , D. A modified iterative closest point algorithm for shape registration [Text] / Dmitrii Tihonkih, Artyom Makovetskii, Vladislav Kuznetsov. ”- Vol. 9971 . ”- [S. l. : s. n.], 2016 . ”- P. 99712D -99712D-8. [12] Miramontes-Jaramillo , D. A Robust Tracking Algorithm Based on HOGs Descriptor [Text] / Daniel Miramontes-Jaramillo, Vitaly Kober , V´ıctor Hugo D´ıazRam // Progress in Pattern Recognition, Image Analysis , Computer Vision , and Applications: 19th Iberoamerican Congress, CIARP 2014 , Puerto Vallarta , Mexico, November 2- 5 , 2014 . Proceedings / Ed. by Eduardo Bayro-Corrochano , Edwin Hancock. ”- Cham : Springer International Publishing, 2014 . ”- P. 54 - 61 . [13] Miramontes-Jaramillo , D. Multiple objects tracking with hogs matching in circular windows [Text] / Daniel Miramontes-Jaramillo, Vitaly Kober, Victor H. Diaz-Ramirez. ”- Vol. 9217 . ”- [S. l. : s. n.], 2014 . ”- P. 92171N -92171N-8. [14] Miramontes-Jaramillo , D. Robust illumination-invariant tracking algorithm based on hogs [Text] / Daniel Miramontes-Jaramillo, Vitaly Kober , V´ıctor Hugo D´ıaz-Ram´ırez . ”- Vol. 9599 . ”- [S. l. : s. n.], 2015 . ”- P. 95991Q -95991Q-8. [15] Miramontes-Jaramillo , D. Real-time tracking based on rotation-invariant descriptors [Text] / Daniel Miramontes-Jaramillo, Vitaly Kober // 2015 International Conference on Computational Science and Computational Intelligence (CSCI). ”- 2015 . ”- Vol. 00 . ”- P. 543 - 546 . [16] Ontiveros-Gallardo , S. E. Objects tracking with adaptive correlation filters and kalman filtering [Text] / Sergio E. Ontiveros-Gallardo , Vitaly Kober. ”- Vol. 9598 . ”- [S. l. : s. n.], 2015 . ”- P. 95980X -95980X-8. [17] Ontiveros-Gallardo , S. E. Correlation-based tracking using tunable training and kalman prediction [Text] / Sergio E. Ontiveros-Gallardo , Vitaly Kober. ”- Vol. 9971 . ”- [S. l. : s. n.], 2016 . ”- P. 997129 -997129-9. [18] Ruchay , A. A correlation-based algorithm for recognition and tracking of partially occluded objects [Text] / Alexey Ruchay, Vitaly Kober. ”- Vol. 9971 . ”- [S. l. : s. n.], 2016 . ”- P. 99712R -99712R-9. [19] Facial recognition using composite correlation filters designed with multiobjective combinatorial optimization [Text] / Andres Cuevas, Victor H. DiazRamirez , Vitaly Kober, Leonardo Trujillo. ”- Vol. 9217 . ”- [S. l. : s. n.], 2014 . ”- P. 921710 -921710-8. [20] Aguilar-Gonza ´lez, P. M. Adaptive composite filters for pattern recognition in nonoverlapping scenes using noisy training images [Text] / Pablo Mario AguilarGonza´lez, Vitaly Kober, V´ıctor Hugo D´ ıaz -Ram´ırez // Pattern Recogn. Lett. ”- 2014 . ”- Vol. 41 . ”- P. 83 - 92 . [21] D ´ ıaz-Ram´ırez, V. H. Object Tracking in Nonuniform Illumination Using Space-Variant Correlation Filters [Text] / V´ıctor Hugo D´ıaz-Ram´ırez, Kenia Picos , Vitaly Kober // Progress in Pattern Recognition, Image Analysis , Computer Vision , and Applications: 18th Iberoamerican Congress, CIARP 2013 , Havana, Cuba, November 20-23 , 2013 , Proceedings, Part II / Ed. by Jose´ Ruiz-Shulcloper, Gabriella Sanniti di Baja. ”- Berlin, Heidelberg : Springer Berlin Heidelberg, 2013 . ”- P. 455 - 462 . [22] Real-time tracking of multiple objects using adaptive correlation filters with complex constraints [Text] / Victor H. Diaz-Ramirez , Viridiana Contreras, Vitaly Kober, Kenia Picos // Optics Communications. ”- 2013 . ”- Vol. 309 . ”- P. 265 - 278 . [23] Diaz-Ramirez , V. H. Target tracking in nonuniform illumination conditions using locally adaptive correlation filters [Text] / Victor H . Diaz-Ramirez , Kenia Picos, Vitaly Kober // Optics Communications. ”- 2014 . ”- Vol. 323 . ”- P. 32 - 43 . [24] Robust Face Tracking with Locally-Adaptive Correlation Filtering [Text] / Leopoldo N. Gaxiola, V´ıctor Hugo D´ıaz-Ram´ırez, Juan J . Tapia [et al.] // Progress in Pattern Recognition, Image Analysis , Computer Vision , and Applications: 19th Iberoamerican Congress, CIARP 2014 , Puerto Vallarta , Mexico, November 2- 5 , 2014 . Proceedings / Ed. by Eduardo Bayro-Corrochano , Edwin Hancock. ”- Cham : Springer International Publishing, 2014 . ”- P. 925 - 932 . [25] Target tracking with dynamically adaptive correlation [Text] / Leopoldo N. Gaxiola, Victor H. Diaz-Ramirez , Juan J. Tapia , Pascuala Garcia-Martinez // Optics Communications. ”- 2016 . ”- Vol. 365 . ”- P. 140 - 149 . [26] Ramos-Michel , E. M. Adaptive composite filters for pattern recognition in linearly degraded and noisy scenes [Text] / Erika M. Ramos-Michel , Vitaly Kober // Optical Engineering. ”- 2008 . ”- Vol. 47 , no. 4. ”- P. 047204 -047204-7. [27] Wu , Y. Object tracking benchmark [Text] / Y. Wu, J. Lim , M. H. Yang // IEEE Transactions on Pattern Analysis and Machine Intelligence . ”- 2015 . ”- Vol. 37 , no. 9. ”- P. 1834 - 1848 . [28] Al-asadi , T. Object detection and recognition by using enhanced speeded up robust feature [Text] / T.A. Al-asadi , A.J. Obaid // International Journal of Computer Science and Network Security. ”- 2016 . ”- Vol. 16 ( 4 ). ”- P. 66 - 71 . [29] Torr , P. H. S. Struck: Structured output tracking with kernels [Text] / Philip H. S. Torr, Sam Hare, Amir Saffari // 2011 IEEE International Conference on Computer Vision (ICCV 2011 ). ”- 2011 . ”- Vol. 00 . ”- P. 263 - 270 . [30] Zhong , W. Robust object tracking via sparsity-based collaborative model [Text] / Wei Zhong // Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). ”- CVPR ' 12 . ”- Washington, DC, USA : IEEE Computer Society, 2012 . ”- P. 1838 - 1845 .