The paper proposes an approach for unauthorized inter-frame video change detection using spatio-temporal slices. This approach can significantly reduce the amount of data processed and replace video processing with image processing that could be performed much faster. To test the efficiency of this approach, we consider a simple algorithm in analyzing adjacent rows of a slice and then classifying the rows based on its result. Experimental studies have revealed that this algorithm shows moderate results in terms of quality, but it has a great potential for improvement, which confirms the prospects of spatio-temporal slices for the given problem.
1.1. Problem statement
Today digital video plays an increasingly important role in the society. In 2015, according to the
Sandvine report [
One of the most common methods of unauthorized video alteration is inter-frame modification, which includes the removal of video fragments or their replacement with copies of other ones. Such changes can remove crime evidence or data about movements of persons or vehicles that are important in a particular context. In video signals obtained from a stationary camera, such changes can be practically invisible. In the case of a moving camera, such changes can also be hard to detect when an intruder cuts or replaces short-term fragments.
In this paper, we consider the problem of detecting inter-frame artificial changes in video signals
taken from a moving or stationary camera. The detection method should work well with video signals
stored in various formats, and also combine high detection accuracy with high speed.
1.2. Review of related studies
In practice, the detection of artificial changes in video can be carried out using digital
forensics methods developed since the second half of the 2000s [
1) camera-based methods that analyze various video artifacts to determine the optical system of the camera; 2) coding-based methods that identify artifacts resulting from encoding video using certain codecs; 3) geometry, or physics-based methods that detect violations in the physical or geometric parameters of the objects observed; 4) pixel-based methods based on detecting changes at the pixel level of the video.
Examples of algorithms aimed to detect inter-frame modifications can be found, in particular, in
[814]. Most of them are coding-based methods [
The coding-based methods [
As for the geometry / physics-based methods, many of them (in particular, [
In this paper, we test a method based on the same idea of detecting deviations in time. However, as
the analyzed data, we propose to use the so-called spatio-temporal slices of video images that are
slices of the video data cube along the time axis and one of the spatial axes (for example, the
horizontal one). If you build several such horizontal slices of video at a certain vertical interval, the
resulting set of images will give quite enough information about object movements, although the data
will have much less volume comparing with the original video. Moreover, for the processing of these
data, we can use computationally effective image processing methods, in particular, parallel-recursive
FIR filters [
In papers [
The paper is organized as follows. Section 2 illustrates the traces of natural events in spatiotemporal slices and outlines the principles for detecting inter-frame video changes with their help. Section 3 describes a simple cut detection method based on these principles. Finally, Section 4 describes the experimental studies.
Figure 1 shows an example of a spatio-temporal slice of a video obtained by a stationary camera. In this figure, the following events occurred are marked with numbers: (1) – movements by the hands of a standing person, (2) – movements of a person who emerged from one door and entered another, (3) – appearance and stopping of a car, (4) - the appearance of a person from the left border of the frame and their movement. As the figure shows, these events lead to the appearance of smooth curves on the cuts characterizing object movements. You can also note the local background shifts due to camera deviations or fluctuations of observable objects (trees, advertising signs, etc.). Figure 2 illustrates four typical types of object movements observed in spatio-temporal slices.
In case of unauthorized video alteration in the time domain, video slices can clearly visualize them by horizontal offsets, as shown in Figure 3. These offsets can appear over the entire frame width. Furthermore, if we consider a set of slices made at different vertical positions, such offsets will be observed in the same lines of the slices corresponding to the same time shift. Thus, we assume that the considered video slices contain enough information to detect inter-frame changes.
(a)
(b) (c) (d) Figure 2. Four types of object movement in spatio-temporal slices: (a) an object enters the camera view and goes beyond it, (b) an object appears in the camera view and goes beyond
it, (c) an object enters the camera view and disappears, (d) an object appears and disappears within the camera view.
Next, we can use the obtained k and k (k ) (for simplicity, we will sign the latest one as k , i.e. without the argument) to detect artificial changes based on the following assumptions. Low values of k and k indicate unaltered videos, while sharp leaps in either k or k may give evidence of artificial changes (see Fig. 4-5).
To detect artificial changes using k and k , we used the algorithm based on supervised learning. It classified slice image rows into two classes: “Cut” and “Non-cut”. To train the algorithm, we used the following features of the rows: p1 k , p2 k , p3 min k k1 , k1 k , p4 min k k1 , k1 k , p5 k med k , p6 k med k . (3) (4) (5) (6) (7) (8)
We should note especially that we calculated the features (3)-(8) in the local maxima only. Function med (x) in equations (7)-(8) means the median of the 4-point neighborhood of pixel x excluding x itself.
To speed up the algorithm, we classified the only rows corresponding to the local maxima of p3 and p4 .
(а)
(b)
Figure 4. Dependence of k (a) and k (b) on k for an unaltered video.
(а) (b)
Figure 5. Dependence of k (a) and k (b) on k for a video with a cut at frame 130.
To test the proposed method, we used two types of video: DVR recordings (made from a moving car), and model records (made mainly by a stationary camera and containing typical pedestrian movements).
To conduct the experiments, we made spatio-temporal slices of the source videos, and then divide them onto 100-line fragments. A half of these fragments were obtained from 100 consecutive frames, while the other half was composite and contained a frame cut in the 50th line. The length of the gap in the frames was a parameter of the experiment. Then for each image, we calculated feature sets. 70% of the data obtained was used as a training set, whereas the remaining 30% was a test sample. When testing the method, it did not use the information about the line of the gap. The experimental studies were carried out in two stages. The first one was aimed to select the most appropriate feature set and classifier model. We considered two models: linear SVM and non-linear SVM with radial basis function. At the second stage, we investigated the algorithm performance for various gap lengths and for different types of video. In addition, at the second stage, we analyzed the efficiency of combining data from different lines of several slices, which was made by summarizing lines.
The first stage of the experiments was carried out on DVR recordings with the gap length of 30 and without slice summation. The results of this stage in terms of classification accuracy (equal to the fraction of the correct classifications) are given in Figure 6. The diagram in Figure 6 shows that the best accuracy values are resulted from the use of the ( p3, p4 ) feature set and linear SVM. Therefore, these options were further used in the second stage (see results in Table 1). The obtained results show that slice summation improves the classification quality sufficiently. We may also notice that the algorithm works better on DVR videos which contain a rapidly changing background. In general, the final results allow us to conclude that the proposed method is able to solve the considered problem, even in the simplified version of the algorithm described in Section 2. 5. Conclusion In this paper, we have tested the approach based on spatio-temporal video slices to solve the problem of detecting unauthorized inter-frame changes in video. This method is theoretically capable to solve this problem with a high speed due to the processing of only a part of the video signal, as well as using fast image processing techniques. To test the efficiency of this approach, we proposed a simple algorithm for detecting inter-frame changes and performed some numerical experiments. Our studies showed that the algorithm provides an accuracy of not less than 0.8 and works better for video captured with a moving camera. The results of the studies allow us to conclude that the method of spatio-temporal image slices looks promising, but the algorithm should be significantly improved in terms of increasing the accuracy at short gap lengths and when using video from a stationary camera. Acknowledgments This work was supported by the Russian Foundation for Basic Research (grants 16-29-09494, 16-41-630676), by the Ministry of Education and Science (grant МК-1907.2017.9), and by the Federal Agency for Scientific Organizations (Agreement 007-GZ/43363/26).