The map constitution task of the customer movement through the store using videoanalitics has been considered. The problem is reduced to the video stream objects detection with further tracking. It is proposed to use pre-trained CNN for object detection. The experimentally justified joint use of pre-trained networks MobileNet-SSD. For the object tracking there have been performed experiments with algorithms built into the OpenCV library: GOTURN, CSRT, KCF, BOOST, TLD, MOSSE, MedianFlow, and MIL. According to the multiple object tracking accuracy (MOTA), the MedianFlow tracker is selected. Experiments were performed using a set of video sequences containing various negative parameters confirmed the effectiveness of the selected solutions.
Today the retail sector is experiencing fierce competition and an increase in consumer demands for
service levels. To maximize the effectiveness of marketing and sales an impressive array of IT
solutions have been offered. One of them is video analytics [
Video analytics using computer vision methods can produce continuous automated data collection, analyzing the sequence of images coming from video cameras in real time or from archival records without additional staff.
According to undertaken studies [
Thuswise, at the moment one of the most promising areas for analyzing customers behavior in a retail store is technology based on video analysis. It allows you to determine customer traffic statistics quickly and effectively, to create a portrait of the target audience, and to study the customer activity.
The basic aim of any trading network is to get the maximum profit. This is achieved by increasing sales (due to an increase in the number of customers) and reducing costs (including reducing the number of staff avoiding the service degradation).
One of the key terms of video analytics in the retail industry is the customer traffic. According to
[
The owner of a point of sale needs to have an up-to-date idea of the institution's attendance, about
the movements of visitors inside the sale area, since this information is used to build a strategy for
attracting and retaining customers, which in its turn is based on:
• optimization of the work of staff (to adjust the number of staff according to the intensity of
the customer traffic in different periods of time);
• sales conversions [
The main source of data for analysis is video cameras located above the entrance and exit, departments of supermarket and cash registers. A generalized plan of sales area is shown in Fig. 1.
The use of video analytics involves automation of four main functions [
Tracking via detection is used to analyze the distribution of customer traffic [
The task of detecting in a video stream should be understood as detecting pre-defined classes of
objects (people, vehicles, furniture, animals and so on) with determining the label and coordinates of
the object's location [
You can represent the object`s location in different ways such as the set of pixels that correspond
to the object [
The input information of the developed system is the video stream S, which is made as a sequence of frames I1, I2, … , Ik, … , IN.
= { ( , ),0 ≤ <
ℎ, 0 ≤ < ℎ ℎ }, = 1, , where width – is the width of frame, height – is the height of frame, Ik(x,y) – is the feature vector of colors, N – is the number of frames; k – is the frame number.
The set containing classes C = {c1, c2, ... , ci, …, cM}. Our task is to detect objects X = {x1, x2, ..., xc}, with their subsequent selection by belonging to a given class (people’s figures): in time by localizing its position on the input sequence of frames.
An object's movement trajectory is a sequence of its positions:
= { ( , , , ℎ, ), +1( , , , ℎ, ), … , + −1( , , , ℎ, )}, where s – is the number of first frame in which the object was detected, l – is the number of frames sequence in which the object is observed, x and y – are coordinates of location; w – is the width; h – is the height and c – is the class number of the object in the video image.
To evaluate tracker accuracy, the MOTA criterion is typically used [
where by the formula:
– is the set of detected objects belonging to the class ci; nc – is the number of detections. The bounding box, which characterizing the location of the object in the image can be represented = { 1( , , , ℎ, ), 2( , , , ℎ, ) … ( , , , ℎ, )}, where (x,y) – are coordinates for each object in the image; (w, h) – is the dimensions of the object, given the width (w) and height (h); c – is the class connected with each bounding box. 3.2.
rectangles that border objects of interest (people figures): corresponding to objects of interest on one frame: where Outk – bounding box, nk – the number of selected objects on a k-frame.
In such a case at the output of the detection algorithm one frame is made asset of dozens bounding In such a case at the output of the detection algorithm we have many bounding boxes = ,1, ,2, … , ,
∈ , { | ∈ }
= { }, = 1, , Tracking of moving objects (tracking) is the creation of a trajectory of movement of target objects (1) (2) (3) (4) (5) (6)
Detection is fundamental and one of the most difficult tasks of computer vision. Deep learning
methods [
Algorithms based on convolutional neural networks (CNN) show the best quality in object
detection tasks. CNN is a special neural network architecture proposed by Yann LeCun which is the
main one used in computer vision [
The classical CNN [
The analysis of Table 1 shows that the R-CNN range models have high accuracy (mAP), but they have a very low video stream detection rate (FPS). YOLO in its turn provides a high FPS value, but has a lower mAP compared to other models. The MobileNet-SSD model has the most optimal parameters in FPS and mAP.
Video surveillance systems use various methods and algorithms to track the object's trajectory. According to [18] they are classified as follows: • Dense optical flow – it helps to estimate the motion vector of all points in the video image, examples of this class of algorithms are Farneback, Horn-Schunck, and also SimpleFlow; • Sparse optical flow – tracks the location of only a few characteristic points (representing the corners or edges of the object) on the video image, an example is the Lucas-Canada algorithm; • Mean-Shift and CamShift determine the locations of the density function maxima; • Kalman Filtering –allows you to get the probable positions of previously found objects in a new frame based on the history of its previous positions. On its basis the best for the date online tracker DeepSort is offered; • Single object trackers (SOT) –assumes that the rectangle selects an object in the first frame and then tracks it in the next frame; • Multiple object trackers (MOT) – assumes tracking multiple objects in each frame; • Tracking algorithms built into the OpenCV library (Boosting, MIL, KCF, CSRT, MedianFlow, Mosse, Goturn, TLD).
Analysis of object tracking methods has shown [19] that at the moment the problem is particularly acute for providing continuous tracking. Most existing tracking systems do not support this functionality or they try to solve the problem by selecting an angle where the probability of overlap is minimal.
One of the main requirements for our task is the ability to track many objects of a video sequence. Also, when choosing a method, you must try to ensure maximum performance and reliability. In this paper trackers of the OpenCV library were subjected to experimental research.
The scheme for obtaining information on the movement of customers based on vid-eo analysis is shown in Fig. 3.
At the first stage the detection algorithm (detector) works. Detection is made on the key frames of the video stream F1, F1+step, F1+2*step, … . In order to check whether new objects have appeared in the frame as well as to see if objects that were lost during the “tracking” stage have appeared, i.e. to correct the tracker operation. The system creates or updates a tracker with new bounding box coordinates for each detected object. where Vi – is the residence time of the object of interest in sells area i.
The result of this work is the construction of the heat map reflecting the areas of interest of customers, the form of which is shown in Fig. 4.
In Fig. 4 the cell at the intersection of date and department the value ′is indicated. The highest value of the coefficient ′corresponds to greater attendance by customers.
There was developed a software implementation of the subsystem in the Python programming language using OpenCV computer vision libraries and Caffe deep learning.
OpenCV [20] is an open source library for computer vision and machine learning. It contains about 2500 algorithms; the main aim of this library is to increase the computational activity of video image processing procedures.
Caffe [21] – a framework that supports the operation of convolutional neural networks. The advantage of using this tool is that it is possible to work in a very simple way and that there are pre-trained models.
Experimental research of the developed video analysis subsystem were carried out on the following computer system configuration: Windows 10 Pro, Intel (R) Celeron CPU N3050 @ 1.60 GHz, 2 GB.
The first step is choosing a detector. Models in the Table 1 are used to detect objects in the image, but they also applicable to the video stream. Since the objects detection in the video stream is reduced to processing a sequence of key frames.
The results of the experiments using standard datasets are given in Table 2.
Analysis of Table 2 showed that the MobileNet-SSD model has the best value (mAP) for the class "Person" and a high rate of speed (FPS). The speed of the detector is important for the tracker. The lower speed of the detector, enhance the risk losing an object of interest during tracking. Thus, the MobileNet-SSD model was selected for the video analysis subsystem.
The choice of tracker was carried out experimentally using 4 test sets. Test datasets are complete video sequences obtained from shopping center surveillance cameras. Datasets contains negative parameters: various lighting, overlapping objects with each other, complete disappearance of the object for some time, changing the size of the object. The presence of such parameters allow you to evaluate the stability of the algorithm to various emergency situations. The characteristics of test datasets are shown in Table 3. The values of the MOTA coefficients for eight trackers on 4 video sequences are presented in Table 5.
According to the made research it can be concluded that the BOOST tracker is very slow (with an average value of 9.75 FPS) and often loses the object of detection. MIL and KCF trackers showed good quality of the algorithm work speed (17.5 and 34 FPS correspondingly. The TLD tracker generates a lot of false positives which makes it unusable.
The MOUSSE tracker provides the highest speed of all the considered trackers (388.75 FPS), and the CRT tracker provides a fairly high accuracy of the tracking algorithm falling short of speed herewith. The MedianFlow tracker has performed well both with regard to speed and accuracy.
In this work, the task of compiling customers movement map through the store was solved. There was performed the analysis of methods for detecting and tracking buyers based on video information. A subsystem for analyzing customer flows using video surveillance has been developed and its software implementation has been completed. At the first stage the problem of detection using CNN is solved. The use of MobileNet and SSD together is well substantiated. There was selected MedianFlow tracker which showed high values of speed and accuracy. The developed set of solutions made it possible to monitor the movements of customers, to identify areas of interest with the goal of further effective personnel management and display of goods.