The paper proposes a method for recognizing smoking event detection from visual data. The method uses a three-dimensional convolutional neural network ResNet, which provides work with video based spatio-temporal features. According to WHO Framework Convention on Tobacco Control [1] there is no safe level of tobacco smoke exposure. Creating a completely smoke-free environment is the only way to protect people from the harmful efects of breathing even second-hand smoke. Human action analysis based on visual processing is significant for many applications such as intelligent video surveillance, analysis of employee and customer behavior. Recognizing a person's smoking while driving can significantly increase road safety [ 2]. To recognize smoking activity on the use of smartwatch sensors as a state-transition model that consists of the mini-gestures handto-lip, hand-on-lip, and hand-of-lip [ 3]. Wu et al. [4] proposed the color-based ratio histogram analysis is introduced to extract the visual clues from appearance interactions between lighted cigarette and its human holder. The techniques of color re-projection and Gaussian Mixture Models enable the tasks of cigarette segmentation and tracking over the background pixels. Smoke detection in the area around human faces and hands can be applied to recognition of the smoking action [5, 6, 7]. The reliable smoke detection is a dificult due to great variability of shape, color, transparency, turbulence variance, non-stable motion, boundary roughness, and time-varying flicker efect in the boundaries of smoke as well as artifacts during shooting such as low resolution, blurring, and weather conditions. The key problem of smoking behavior recognition is the irregular shape: diferent ways to hold a cigarette, types of tobacco products, bad weather and shooting conditions.
In this paper spatio-temporal features based smoking activity detection algorithm, which allows recognizing human smoking activity regardless of the person’s appearance, the way to hold a cigarette, the type of cigarette, the distance of the object of interest, and movement patterns. pause, falls down, pause, rises again; ∙ lighting a cigarette: ∙ lip movement on close-up scenes;
Smoking activity belongs to a group of atomic actions that can be recognized only if there is a certain set of spatio-temporal features. Four atomic action groups are considered: ∙ arm position changes. The sequence of actions: the hand rises to the level of the lips, times); ∘ tilt of the head; ∘ using a cigarette lighter, using a lighter involves a sequence of actions: — bringing the lighter to the face with one hand; — the thumb of this hand starts the mechanism (the action can be repeated several following actions: — the other hand can prevent the cigarette from fading and block the view to recognize previous actions (in this case, both hands are at the level of the lips); — the hands are lowered; ∘ lighting up with matches, the use of matches for lighting cigarettes consists of the — the cigarette is clamped between the teeth; — both hands are at chest level or just below the chest; — one hand is performed with a small wave (the action can be repeated); — one hand remains at chest level, the second changes position, moving higher to the chin or lips, — a wave of the hand (to extinguish the match); — lowering the hands; ∙ flicking the ash from the cigarette (the action may not be present in the frame) consists of the following steps: the withdrawal of the hand with the cigarette down and the characteristic movement of the hand or fingers of the hand.
Smoking activity recognition is implemented using a three-dimensional neural network based on the spatio-temporal features in the entire video data.
Visual information as a result of real-time video shooting may include objects with dynamic behavior, noise of the hardware or transmission lines, as well as artefacts afected by weather conditions (for example, rain or snow, poor luminance in the morning or evening). Because of this, the quality of smoking action recognition significantly degrade. Therefore, scaling and mean subtraction [8] are used to solve this problem. To implement preprocessing algorithms, a computer vision library OpenCV (Open Source Computer Vision Library) was used [9]. Thus, the video sequence preprocessing is performed according to the expressions: = − , =
, = −
,
(
AlexNet [10], VGG [11] and ResNet [9] neural networks are most often used to classify images and video sequences. The ResNet neural network is fully convolutional, so it is used for space-time volume extraction, unlike many architectures with fully connected layers, including AlexNet and VGG-16, which contain several levels of the maximum pool that can damage the actions evaluation. The ResNet network contains only one pool level immediately after the conv1 layer. The reduced number of bonding layers makes ResNet more suitable for visual recognition of smoking, since spatial details must be preserved to recognize this process.
In the work the 34 layers ResNet neural network was used that shows computational eficiency in solving classification problems [ 12]. In order to use ResNet to estimate multi-frame optical lfow, it is necessary to extend this architecture, replacing all × two-dimensional convolutional kernels with an additional time dimension × × 3, as described in article [13]. The pool layers in the decoder are expanded in a similar way. The neural network transformed in this way in the paper is called ResNetM, its composition is presented in Table 1.
In Table 1 the residual blocks are grouped in square brackets. Batch normalization is used after each convolutional layer. The main diference between this architecture and ResNet is the use of 3D kernels and a modified downsampling operation, whereby feature maps in the convolution layer are combined with several adjacent frames in the previous layer, thereby capturing motion information. inputs is performed periodically in steps of 2.
The dimensions of the convolutional kernels are 3 × 3 × 3. The network uses 16-frame RGB clips as inputs. The dimensions of the input clips are 3 × 16 × 112 × 112. Downsampling of
Core 7 × 7
× 7 3 3 3 3 ︂[ 3 × 3 ︂[ 3 × 3 × 3 × 3 ︂[ 3 × 3 ︂[ 3 × 3 × 3 × 3 ︂] ︂] ︂] ︂] × 3 × 3 × 3 × 3 × 3 × 4 × 3 × 3 × 6 × 3 × 3 × 3 64 64 128 256 512
The proposed method uses deep learning network for smoking action detection by recognizing actions that are characteristic of a person who is in the process of smoking. The block diagram of the smoking activity detection algorithm is shown in Figure 1.
Stochastic Gradient Descent (SGD) with momentum is used to train the neural network. Training samples are randomly generated from the videos in the training set. Time positions are selected evenly. Next, 32-frame clips are set around the specified time positions. If the video is shorter than 32 frames, it will loop as many times as necessary to reach the set duration. Then the spatial positions are randomly selected from four corners or one center. In addition to the positions, the spatial scales of each sample are also specified for multiscale cropping. The frame is cropped at the time-space positions. The size of each sample is 3 channels× 32 frames× 112 pixels× 112 pixels, and each sample is flipped horizontally with a 1/2 probability. It also subtracts the average of our dataset from the sample for each color channel. All created samples retain the same class labels as their original videos. Model training uses cross entropy as a function of loss. The training parameters include a damping of 0.001 and 0.9 for the impulse. The learning rate is 0.1 and divided by 10 after saturation of the validation loss. When fine tuning is performed at a learning rate of 0.001, the scale attenuation is 1e − 5.
At the first stage, the neural network is initialized, the parameters are set, and after that the
video sequence is fed to the input. Initialization of the classes is performed, which allows the
classification of the dataset: “smoking”, “no smoking”. The duration of the sample is determined,
that is, the number of frames for classification is 32, and the spatial sizes of the sample are
112 × 112. To create input clips, the sliding window method is used, in which only the oldest
frame in the list is discarded, making room for the newest frame. Each video is then split into
non-overlapping 32-frame clips. This operation occurs using a loop that reads frames from the
video stream, then checks for frame capture. If a frame is captured, then each clip is cropped
around the center position at the maximum scale, an average subtraction is performed and a
new frame is added to the queue, otherwise the loop exits. The new cycle allows you to check if
the queue is full. At the end of this cycle, a blob object is created. A “blob object” or “blob” is
a collection of frames with the same spatial dimensions, expressed in width and height, and
the same depth, that is, the number of channels that must be preprocessed in the same way. A
blob object has the following dimensions: (
Next, in order to extract the space-time characteristics, each instance is transmitted through a 3D convolutional neural network. Smoking is recognized by finding multiple optical flow. The optical flow is calculated at each point, then a motion map is formed. Each feature map of a convolutional layer is associated with several consecutive adjacent frames in the upper layer. The next step is to assess the probability of smoking in the clips. The network “scans” the sequence of thirty-two frames, generates motion paths, analyzes the similarity to a known smoking pattern, and finds the probabilities of smoking in each frame, which are then averaged over all clips. The class that has the highest score indicates the action in the given video sequence. If the probability is greater than or equal to 0.5, then smoking in these frames is recognized.
In order to the video-based smoking detection model work, the following specifications are
required: a minimum of 2GB NVIDIA graphics card and installed software: CUDA and cuDNN.
The model uses Anaconda and Python packages including OpenCV, matplotlib, and Pytorch.
Experimental studies were carried out with the characteristics of a laptop Intel (R) Core (TM)
i76700HQ processor, 2.60 GHz processor clock, 8 GB RAM, Windows 10 operating system, NVIDIA
GeForce GTX 960M graphics processor, 2 GB dedicated graphics processor memory. The
modified neural network was trained on 6766 videos from the HMDB51 dataset [ 14]. The video
shows actions that can be grouped into five groups:
(
Actions of categories (
Tables 3 and 4 shows the frames of some of the video sequences used and the results of smoking recognition. The results of the smoking recognition method are marked with the labels “smoking” — “no smoking”.
The test video data is supplemented with videos in which the action is visually similar to smoking, but, thanks to the spatial and temporal features of the neural network and the identified pattern of characteristic smoking movements, it is able to distinguish these actions from smoking action. In video 9 a girl eats a lolipop; video 11 a girl bites a pen; video 15 a man eats ice cream. The training sample was 80%, the test sample was 20% of the total sample. To evaluate the efectiveness of human smoking activity detection and recognition algorithms, the indicators of detection accuracy (TR), false-positive (FAR) and false-negative (FRR) were used. The results of smoking detection for neural network architectures ResNet and modified network ResNetM are shown in Table 5.
Era
0.2024 0.2058 0.2448 0.2259 0.2267 0.6699 0.7346 0.7613 0.7984 0.7984 0.9198 0.9280 0.9095 0.9280 0.9125
Resolution: 1280× 720.
Resolution: 270× 360.
Resolution: 1280× 720.
Resolution: 480× 360.
Resolution: 1280× 720.
Experimental studies conducted on 20 video sequences obtained in real-world shooting conditions confirm the eficiency of the proposed method for recognizing smoking. The ResNet neural network architecture, modified to a three-dimensional neural network, ensures that the spatial-temporal signs of smoking are taken into account and shows, on average, 15% higher accuracy in recognizing the smoking actions compared to the basic architecture. The developed software implementation of the smoking recognition method provides real-time operation.