=Paper=
{{Paper
|id=Vol-2870/paper130
|storemode=property
|title=Application of Innovative Approaches to Video Segmentation in a Criminal Process
|pdfUrl=https://ceur-ws.org/Vol-2870/paper130.pdf
|volume=Vol-2870
|authors=Lidia Moskvych,Yuliia Riepina,Kostyantyn Shcherbinin
|dblpUrl=https://dblp.org/rec/conf/colins/MoskvychRS21
}}
==Application of Innovative Approaches to Video Segmentation in a Criminal Process==
https://ceur-ws.org/Vol-2870/paper130.pdf
Application of Innovative Approaches to Video Segmentation in
a Criminal Process
Lidia Moskvycha, Yuliia Riepinab and Kostiantyn Shcherbininc
a
Yaroslav Mudryi National Law University, 77 Pushkinska str., Kharkiv, 61024, Ukraine
b
Academician V. V. Stashis Scientific Research Institute for the Study of Crime Problems of the National
Ukrainian Academy of Law Sciences, 49 Pushkinska str., Kharkiv, 61002, Ukraine
c
Self-Employed IT Professional, 240-B, Saltivske hwy., Kharkiv, 61171, Ukraine
Abstract
Video data segmentation, processing, analysis and indexing of segmentation results in time
and space are proposed to improve the efficiency of systems for metric search and recognition
of dynamic visual information in databases with a query `ad exemplum`; declared the
possibility and usefulness of using video segmentation, based on the analysis of
multidimensional time series, in the criminal process, emanated on the need to introduce
innovative technologies into the criminal process.
Keywords1
Video surveillance, criminal procedure, innovative technologies in criminal procedure, video
segmentation, video data search.
1. Introduction. Formulation Problem in General
Protection of a person, society and the state from criminal offenses, protection of the rights, freedoms
and legitimate interests of participants in criminal proceedings, as well as ensuring a prompt, complete
and impartial investigation and judicial investigation so that everyone who committed a criminal
offense is brought to justice in moderation of his guilt, not a single innocent person was accused or
convicted, not a single person was surrendered to unreasonable procedural compulsion and that due
process is applied to each participant in criminal proceedings, is the task of the criminal procedure [1].
Video surveillance in public places is becoming more widespread. The rapid development of
technology and the growing sense of insecurity among the population has gradually led the population
to perceive video surveillance as a useful tool in the context of crime prevention and detection [2].
Indeed, the use of video surveillance allows you to record the events taking place objectively, unlike a
person, whose perception of information is subjective. The law [1] defines evidence in criminal
proceedings as factual data obtained in the manner prescribed by it, on the basis of which the
investigator, prosecutor, investigating judge and the court establish the presence or absence of facts and
circumstances relevant to criminal proceedings and are subject to proof. Among the procedural sources
of evidence, the Law also names documents - material objects specially created for the purpose of
preserving information, containing information recorded with the help of, including images,
information that can be used as evidence of a fact or circumstances established during criminal
proceedings. Such documents may include video recording materials, including electronic ones.
________________________
1
COLINS-2021: 5th International Conference on Computational Linguistics and Intelligent Systems, April 22–23, 2021, Kharkiv, Ukraine
EMAIL: moskvichlida@gmail.com (L. Moskvych); riepina.yuliya@gmail.com (Yu. Riepina); k.s.shcherbinin@gmail.com (K. Shcherbinin)
ORCID: 0000-0001-7339-3982 (L. Moskvych); 0000-0002-3157-3181 (Yu. Riepina); 0000-0001-7974-9361(K. Shcherbinin)
© 2021 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
Currently, the law enforcement system of Ukraine is undergoing reforms that are impossible without
considering the use of modern information technologies. The use of video surveillance in law
enforcement can increase the efficiency of public order protection and public safety [3].
Object detection is a key module in most visual filming, video surveillance and security
applications.
The intensive development and widespread dissemination of information and communication
technologies in the modern world aggravated by the complication of social relations in the
direction of political and economic unification and diversification determine the modernization of
society, which is manifested in the digitalization of social relations and management processes,
as well as in the creation of a single information space on the basis of the Internet network [4].
In navigation and tracking systems, the proposed approaches for analysing video sequences can be
used to obtain data on scene changes, in particular, this concerns issues of video surveillance over a
certain area. The approaches, proposed in this work, make it possible to record and save only that
information that is "essential", i.e. the one that occurs in the event of a significant change in the video
data. Thus, it is supposed to increase the efficiency of search and recognition of visual information
necessary for solving tasks of criminal proceedings.
2. Related Works
Valid search for video data by meaningful criteria, in particular, with ‘ad exemplum’ queries, is
provided by the development of methods and models for indexing video data, the synthesis of measures
and metrics for comparing events, plots, scenarios, the implementation of relevant feedback in order to
iteratively clarify the information needs of the User.
Interdisciplinary multifactorial integration of methods of pattern recognition, computer vision,
database management, analysis of multidimensional time series, artificial intelligence in general is the
way to create such systems. The main influence on the development of methods for processing visual
information that determine the tools of such systems was exerted by Ukrainian and foreign scientists
S.G. Antoshchuk, A.M. Akhmetshin, E.V. Bodyansky, H. Burkhardt, R.M. Haralick, Z. Hu, G. Liu, ,
V.P. Mashtalir, S. V. Mashtalir V.N. Krylov, D.D. Pieleshko, E.P. Putyatin, M.I.Shlesinger, T.E. Rak,
M. Sonka, R.A. Vorobel, P. Zezula, G.N. Zholtkevich and others.
Analysis of the state and trends in the development of methods for searching for visual information
with query `ad exemplum` allows us to assert that, despite numerous studies in this direction, the growth
of the accumulated video data and the intensification of their use require the creation of new high-speed
valid search tools.
A certain perspective is the carrying out of theoretical and experimental studies on the development
of models and methods for ensuring increased performance of metric search based on two-stage video
segmentation: in time to obtain homogeneous (in a broad sense) video segments and in space (in the
field of view) to obtain multidimensional time series, analysis which creates the prerequisites for video
indexing.
Thus, the study of the approaches of segmentation of multidimensional time series induced by the
segmentation of individual video frames, as well as the identification of necessary and sufficient
conditions that can significantly speed up the search, is an urgent task.
We believe that the prospect of using innovative approaches in criminal proceedings proposed by
the authors deserves the interest of appropriate proficient for further studies.
3. Methods
The research used the basic provisions of the mathematical apparatus for pattern recognition,
artificial neural networks (ANN), set theory, algebra, elements of mathematical statistics.
4. Formulation of Goals
The purpose of the study was to develop approaches for video data segmentation, processing,
analysis and indexing of segmentation results in time and space to improve the efficiency of metric
search and recognition systems for dynamic visual information in databases with queries based on the
pattern, and writing an article is an attempt to declare the possibility and usefulness the use of video
segmentation based on the analysis of multidimensional time series in the criminal process, based on
the need to introduce innovative technologies into the criminal process.
5. Main Part
5.1. Detecting property changes in multidimensional time series based on
custom model
Let us assume that in some feature space there are a multidimensional time series [5], that
characterize the video x(k) = (x(k), x (k) ,..., x(k))T, where 1 £ k £ N – is discrete time, at moments
which observations are made. The segment of this series S(a, b) = {a £ k £ b} is a statistically
homogeneous sequence x(a), x(a+1), ..., x(b), and the problem of time segmentation is a search c
disjoint segments Sc={Si(a, b), 1 £ I £ c}, such that что a1 = 1, bc = N, ai = bi-1 + 1. So, it is necessary
to find c intervals S1 < Sa < ... < Sc and their boundaries ai, bi.
Consider a sequential, on-line implementation-oriented detection of changes in the properties of
multidimensional time series in the process of adaptive identification of the VAR process model, which
in the general case connects past and current observations x(k) in the form:
𝑥(𝑘) = 𝐵! + ∑$%&! 𝐵# 𝑥(𝑘 − 𝑙) + 𝜉(𝑘) (1)
where B0 ={b0i}-(n´1) - vector of mean values,
Bl = {blij }–(n´n) - parameter matrices,
p – model order.
It should be emphasized that the initial information for solving the problem of identification and
detection of changes is only the n-dimensional time series x(k) itself, the values of which arrive
sequentially in time.
To simplify further calculations, we introduce into consideration the composite ones:
matrix B = (B0 ⋮ 𝐵' ⋮ . . . ⋮ 𝐵( ) and vector X(k)=(1,xT(k-1),..., xT(k-p))T dimensions (n´(pn+1)) and
((pn+1)´1) respectively, after which we rewrite equation (1) in the form
x(k ) = BX (k ) + x(k ) , (2)
where the matrix of a priori unknown parameters B contains practically all the necessary information
about the properties of the controlled signal.
The identification problem is that in accordance with the real signal (2) a tunable model
xˆ(k) = B (k - 1) X (k) , (3)
the matrix of parameters B(k) of which is refined at each time step k by minimizing the accepted
identification criterion, which is a certain function of the difference between the calculated xˆ(k) and
experimental data x(k). So, the synthesized model (3) must be workable in the forecasting mode, and
the violation of predictive properties can be a sign of the occurrence of certain recurrent procedures,
which can be presented in a generalized form [6]:
𝐵(𝑘) = 𝐵(𝑘 − 1) + 𝛾(𝑘)𝑒𝑋 ) (𝑘);
I (4)
𝑒(𝑘 ) = 𝑥(𝑘 ) − 𝑥N(𝑘) − 𝐵(𝑘 − 1)𝑋
where g(k) is the scalar or matrix gain of the algorithm, which determines its properties and depends
on the adopted identification criterion; e(k) - vector identification error.
In practice, the most widespread algorithms are those associated with the criterion for the minimum
sum of squares of identification errors
𝐼(𝑘) = ∑,-&' 𝛽(𝑢) ∥ 𝑒(𝑢) ∥* = ∑,-&! ∑$%&' 𝛽(𝑢)𝑒+ (𝑢)* (5)
and its modifications determined by the adopted system of weights b(u). The well-known is the least
squares method, in which all weights have the same weight, that's
𝐼(𝑘) = ∑"#$% ∥ 𝑒(𝑢) ∥! . (6)
The algorithm corresponding to (4), (6) has the form:
&(")) * (")*("+%)
𝐵(𝑘) = 𝐵(𝑘 − 1) + ;
%,) * (")*("+%))(")
, (7)
*("+%))(")) * (")*("+%)
𝑃(𝑘) = 𝑃(𝑘 − 1) − %,) * (")*("+%))(")
and with constant parameters (2) provides monotonic convergence of estimates B(k) to the true values
of parameters B. We assume that the recursive least squares method is not suitable for detecting changes
in properties. The choice to (6), (7) is the one-step procedure
.(,)1 ! (,)
𝐵(𝑘) = 𝐵 (𝑘 − 1) + 1 ! (,)1(,), (8)
generated by the one-step identification criterion
I(k)= ||e (k)||2 (9)
and is a generalization of the Kachmazh algorithm [7] to the vector-matrix model (3). Having a high
speed, procedure (8) does not have filtering properties, and therefore is not able to distinguish between
changes in the signal and the influence of the stochastic component x(k ).
In this regard, it seems expedient to use finite memory algorithms that have both smoothing and
tracking properties, the compromise between which is set by the size of the memory. Loss of predictive
properties of the model (3) and the need to rebuild the memory of the algorithm can serve as a sign of
emerging changes.
Returning to criterion (5), we note that in the class of algorithms generated by it, the exponentially
weighted least squares method with the criterion
𝐼(𝑘) = ∑,-&' 𝛽,2- ∥ 𝑒(𝑢) ∥* (10)
and a recurrent setup procedure
.(,)1 ! (,)3(,2')
𝐵(𝑘 ) = 𝐵 (𝑘 − 1) + 451 !(,)3(,2')1(,) ;
S (11)
' 3(,2')1(,)1 ! (,)3(,2')
𝑃(𝑘) = 4 (𝑃(𝑘 − 1) − 451 !(,)3(,2')1(,) ,
where 0 < b £ 1 is the smoothing parameter.
At this point, it should be noted that an identifier with exponential smoothing is generally unstable,
which leads to an “explosion of parameters” of the covariance matrix, which occurs especially often at
high dimensions of the processed signal x(k). Thus, using the traditional exponentially weighted
recurrent least squares method is complicated by the poor conditioning of the information matrix
∑,-&' 𝛽,2- 𝑋(𝑢)𝑋 ) (𝑢), (12)
generated by a high level of correlation between the components xi (k).
Replacing the operation of inversion of the weighted information matrix (based on the Sherman-
Morrison formula) with the operation of pseudo-enrichment using Greville's theorem solves the
problem [8]. However, the new algorithm is too cumbrous from a computational point of view,
especially for large n.
As a result of that, should pay attention to the use of a multidimensional modification of the
exponentially weighted stochastic approximation algorithm [9] in the form:
.(,)1 ! (,)
𝐵(𝑘) = 𝐵(𝑘 − 1) 46(,2')5∥1(,)∥$
U ; (13)
𝑟(𝑘) = 𝛽𝑟(𝑘 − 1)+∥ 𝑋(𝑘) ∥* ,
which is a kind of compromise between procedures (8) and (11) and has the necessary smoothing and
tracking properties.
In [10, 11], a method for regulating the smoothing parameter b is proposed, based on the control of
statistics characterizing the prediction error of a one-dimensional signal. It is assumed that the tuning
of the model parameters is performed using the exponentially weighted Kalman-Main algorithm, which
for the i-th component xi(k) can be written as:
(.(,)1 ! (,)3% (,2')%
𝑏% (𝑘) = 𝑏% (𝑘 − 1) + ;
48%$ 51 ! (,)3% (,2')1(,)
S (14)
' 3 (,2')1(,)1 ! (,)3% (,2')
𝑃% (𝑘) = (𝑃% (𝑘 − 1) − % $ ! (,)3 (,2')1(,) ,
4 48% 51 %
where bi(k) – i-th matrix row B(k),
(o)i – i -th the row of the corresponding matrix product.
If the variances of individual components xi(k) of the disturbance vector are unknown, then in (14)
an estimate can be used in the form:
𝜎%* (𝑘) = 𝜎%* (𝑘 − 1) + 𝑃% (𝑘 − 1) ]𝜎%* (𝑘 − 1) − 𝑒%* (𝑘)^ ;
[ ' 3%$ (,2')
(15)
𝑃% (𝑘) = 4 (𝑃% (𝑘 − 1) − 453 (,2')
,
%
It is also assumed that the model quite accurately describes the controlled signal at time intervals s
of observations, and the parameters can change in jumps at arbitrary times ka. The variable value b(k)
is regulated by statistics
. $ (-)
𝑇% (𝑘) = ∑,-&,29 4(,2')8$ 51 !% (-)3 (-2')1(-), (16)
% %
having 𝑥 * distribution with s degrees of freedom, while b(0)=1. Regulation b(k) is performed at
discrete times ts according to the following rule:
1 𝑎𝑡 𝑘 < 𝑠, 𝑘 = 𝑡𝑠 𝑎𝑛𝑑 𝑇% (𝑘 ) ≤ 𝑥:* ;
𝛽(𝑘) = [ 𝛽(𝑘 − 1) − ∆𝛽 𝑎𝑡 𝑘 = 𝑡𝑠 𝑎𝑛𝑑 𝑇% (𝑘) > 𝑥:* ; (17)
𝛽(𝑘 − 1) 𝑎𝑡 𝑡𝑠 < 𝑘 < (𝑡 − 1)𝑠, 𝑡 = 1, 2, …
*
Here 𝑥: is the quantile of the law 𝑥 * , corresponding to the significance level j , Db is the regulation step.
Rule (17) provides for a change in b(k) for values of k, that are multiples of s (for intermediate k,
the value of b(k) remains unchanged), the fact of changes is recorded at the moment the second relation
(17) is realized.
The cumbersomeness and inertia of this procedure forces us to look for other faster and more effective
methods of detecting changes. So, in [12], a method for regulating the smoothing parameter b based on
the Mann-Whitney criterion was proposed. In this case, the controlled characteristic is the value
∑,-&,295' 𝑠𝑖𝑔𝑛(𝑥% (𝑢) − 𝑥N% (𝑢)) ≥ 𝛾, (18)
where γ – some threshold value,
s - the size of the sliding control window,
0 𝑎𝑡 𝑥% (𝑢) = 𝑥N% (𝑢);
𝑠𝑖𝑔𝑛p𝑥% (𝑢) − 𝑥N% (𝑢)q = [ +1 𝑎𝑡 𝑥% (𝑢) > 𝑥N% (𝑢); (19)
−1 𝑎𝑡 𝑥% (𝑢) ≤ 𝑥N% (𝑢).
The control process begins with the value b(1)=0, which corresponds to the maximum speed of
algorithm (13). The exponentially weighted recursive least squares method in this situation, of course,
is fundamentally inoperable. During the authentication may arise:
∑,-&,295' 𝑠𝑖𝑔𝑛(𝑥% (𝑢) − 𝑥N% (𝑢)) < 𝛿, (20)
which means the dominance of the stochastic component xi of the signal xi over the “drift” one. In
this case, it is necessary to improve the smoothing properties of the algorithm, i.e. increase it according
to the rule: b(k) = b(k -1) + Db.
In situation (18), the drift component of the signal prevails and the algorithm does not have time to
track the changes that have arisen. In other words, it is worth to decrease storage b(k) = b(k -1) - Db
and record the fact of changes.
To control changes in a multidimensional time series, it is proposed to use a modification (18) in the
form:
𝑚𝑎𝑥% (∑,-&,295' 𝑠𝑖𝑔𝑛(𝑥% (𝑢) − 𝑥N% (𝑢))) ≥ 𝛾, (21)
Actually, not statistical, but heuristic procedures e. g. the methods of Chow, Brown, Trigga-Leach,
Shawn, etc. have become more widespread [12, 13, 14, 15], based on some heuristics, and therefore
bearing an element of subjectivity. Given the shortage of a priori and current information on the
characteristics and properties of the monitored signal, preference should naturally be given to these
methods. The basis of most heuristic methods is as follows: a set of values for the smoothing parameter
b, is specified, for example, 0; 0.05; 0.1; ... 0.95; 1 and a set of characteristics that determine the quality
of identification. Most often these are:
- is the current estimation error of the i-th component: ei (k , b)=xi(k) - 𝑥N% (k , b);
- the cumulative sum of errors: Gi (k, b)=ei (k, b)+Gi(k -1, b);
- mean absolute error: di (k, b)=(1-b)ei (k, b) +bdi (k-1, b);
- average error: 𝑒̅% (𝑘, 𝛽) − (1 − 𝛽)𝑒% (𝑘, 𝛽) + 𝛽𝑒̅% (𝑘 − 1, 𝛽);
. (,,4)
- relative mean error: 𝑒̃% (𝑘, 𝛽) = (1 − 𝛽) <% (,) + 𝛽𝑒̃% (𝑘 − 1, 𝛽);
%
- root mean square error: 𝑒̅%* (𝑘, 𝛽) = (1 − 𝛽)𝑒̅%* (𝑘, 𝛽) + 𝛽𝑒̅%* (𝑘 − 1, 𝛽).
The smoothing parameter b should be adjusted accordingly. The adoption of such a decision is
associated with changes, that have occurred as a result of exceeding a certain threshold γ value of the
selected controlled characteristic.
A stationary stochastic signal is usually associated with a value of b lying in the interval 0,7 £ b £
0,9 [15]. The simplest form of control is that when the value of е{i(k, b) for some b exceeds the threshold
0.05 [13], the smoothing parameter decreases according to the rule: b(k) = b(k - 1) - D b, and the
identification process continues with a new b(k). If the value of b exceeds the threshold of 0,7 £ b(k)
£ 0,7), a decision is made on the occurrence of a change in the signal.
The Chow method adjusts three models simultaneously with the values of the smoothing parameters
b, b+Db and b-Db. It seemed to be more effective, albeit more complicated.
The Chow method is more effective, but also more complex. Here, three models with the values of
the smoothing parameters b, b+Db and b-Db are simultaneously adjusted. For example, the best result
is obtained, when the current time k from b(k)=b+Db, then on the next time step, a new triple of
smoothing parameters b, b + Db and b + 2Db is used. In the case, when the best model is achieved at
b(k)=b-Db, after a triple b, b-Db, b-2Db are formed, finally, in the case, when the best result is obtained
at b(k)=b, the set b, b+Db, b-Db are preserved.
Simpler and more efficient methods are those using a tracking signal, which is an indicator of
changes in monitored signals [14, 15]. There are a number of forms of the tracking signal, while its
going beyond certain limits indicates changes that have occurred.
R. Brown proposed to use the expression [16] as a tracking signal:
∑&
'() .% (-)
𝑇%= (𝑘) = (22)
?8%$ (,)
the physical meaning of which is that if the tuned model is adequate to the controlled signal, the sum
of errors varies around 0, while not exceeding some boundaries that are set a priori for a given level of
probability for a certain variance of the sum of forecast errors, which tends to the value
'
lim 𝜎%* (𝑘) = '2('24)$(+,-)) 𝜎%* . (23)
,→A
D. Trigg, A. Leach suggested using the ratio [17]:
) ` (,)
𝑇%)B (𝑘) % , (24)
C% (,)
where 𝑇%` (𝑘) = (1 − 𝛽)𝑒% (𝑘) + 𝛽` 𝑇%` (𝑘 − 1)
In other words, this is not the total amount of deviations, but a smoothed error, and the inequality
must be follow b¢£b.
At b¢=b the tracking tone will vary between –1 and +1. To introduce automatic feedback, D. Trigg,
A. Leach suggested to computating the smoothing parameter therefore the relation b(k) =1- |TTL(k)|
and fix the discrepancies in the signal with significant changes in b(k).
Mention should be made of Shaun's modification [15]: b(k) =1- |TTL(k -1)|.
A growing of the tracking signal indicates an increase in the discrepancy among the model and the
controlled sequence, to compensate for which, a faster response of the identifier is required, which is
provided by a lower value of the smoothing parameter. This provides negative feedback.
In [13] it is emphasized that in identifying processes with a sufficiently smooth drift, Brown's
method has advantages. Sharp spikes are better identified with the Trigga-Leech tracking signal. Just
like that this form maybe useful when analysing changes in a time series.
5.2. Detect property changes in multivariate time series based on
exponential smoothing
The main disadvantage of the approach to detecting changes considered above is a significant
number of parameters of the tunable model (3), which is n(pn+1), which can cause certain difficulties
at large n and high frequencies of information arrival for processing. A fairly convenient mathematical
tool for solving the problem of detecting changes in the properties of one-dimensional stochastic
sequences is exponential smoothing [14, 15, 16], the essence of which can be illustrated by the
following example.
Figure 1: Discontinuous changes in the mean level of the time series [18]
Let us introduce an elementary model of the form:
xi (k)=bi +xi(k) (25)
and suppose that the coefficient bi can change abruptly from time to time as shown in Figure 1.
Worth mentioning about the magnitude and time of change of the coefficient bi are a priori unknown,
and the time interval b -a, during which the value of the coefficient bi remains unchanged, significantly
exceeds the signal quantization step (the time between two consecutive observations).
Since exponential smoothing is designed to solve the forecasting problem, we first consider the
problem of constructing a forecast 𝑥N% (k+1) at time k. Let us also assume that the parameters of model
(25) at the moment k +1 coincide with those that we received by the moment k. In this case, the problem
is reduced to estimating the current value bi(k ) from k previous observations. Since the value of bi
changes over time, then to obtain this estimate, observations xi(k),xi(k-1),... should be taken with a
greater weight than observations obtained much earlier. In principle, this problem can be successfully
solved using the simplest sliding window method; moreover, if the size of this window s is given, then
the smoothed estimate has the form:
'
𝑥̅% (𝑘) = 9 ∑,-&,295' 𝑥% (𝑢). (26)
The reaction rate of an identifier using a sliding window apparatus to a change in the signal depends
on the value s of the averaged recent observations. This speed increases with decreasing s and vice
versa. On the other hand, a decrease in s in the absence of data changes decreases the accuracy of the
resulting estimate bi(k) = 𝑥̅i(k), since its variance can be written as:
8$*
𝜎+% = 9% . (27)
This shows that the accuracy of the bi(k) estimate and the response rate of the identifier are
contradictory requirements. Taking into account the obvious relation
< (,)2<% (,29)
𝑥̅ % (𝑘) = 𝑥̅% (𝑘 − 1) + % ,
9
or
𝑠% (𝑘) = 𝛼𝑥% (𝑘) + (1 − 𝛼)𝑠% (𝑘 − 1) = 𝛼𝑥% (𝑘) + 𝛽𝑠% (𝑘 − 1), (28)
where procedure (28) is the traditional exponential smoothing.
In expression (28), to distinguish exponential smoothing from the moving average, the notation si(k)
is introduced instead of 𝑥̅i(k). The quantity a, which is analogous 1‚𝑠 to the moving average,
determines the weights of the observations present in the smoothed estimate.
From expression (28) it follows that the current value of the smoothed value si(k) is equal to its
previous value plus some fraction of the difference between the current observation and the previous
value of the smoothed value. Since operation (26) is implemented for all observations of the time series,
it can be rewritten considering all previous observations in the form: 𝑠% (𝑘) = 𝛼𝑥% (𝑘) +
+ (1 − 𝛼)(𝛼𝑥% (𝑘 − 1) + (1 − 𝛼)𝑠% (𝑘 − 2)) = 𝑥% (𝑘) + (1 − 𝛼)(𝛼𝑥% (𝑘 − 1) + (1 − 𝛼)p𝛼𝑥% (𝑘 −
− 2) + (1 − 𝛼)𝑠% p𝑘 − 3))) = ⋯ = 𝛼 ∑,2' - ,
-&!(1 − 𝛼) 𝑥% (𝑘 − 𝑢q + (1 − 𝛼) 𝑥% (0q.
Thus, the value si(k) is a certain combination of all previous observations, the weight of which
decreases exponentially with time. The current observation has weight a, the value of which lies in the
interval [0,1]. The limiting value a=0 corresponds to the case s = ¥ in the moving average. Moreover,
si(k) = si(k -1), that is, the si value is independent of new information. The limit value a=1 means that
*
the background does not affect the current rating at all, i.e. 𝑠% (𝑘) = 𝑥% (𝑘), аnd 𝜎+% = 𝜎%* .
Thus, the accuracy and speed of the response of the identifier to the rate of change in the signal
completely depends on the accepted value a. A small value of provides a greater accuracy in estimating
bi at a stationary signal, but a slow response to changes, while an increase a in will increase the rate of
this reaction.
Usually [16], the value of a is in the range from 0.01 to 0.3, and the number s is in the range from 6
to 200. Since this range is large enough, in each specific problem this parameter must be selected in a
special way. The need to use large values of a (small values of s) indicates a discrepancy between the
selected and real models of the monitored signal, i.e. can serve as a sign of a change in its properties.
It was noted above that with exponential smoothing, the weight of the current observation has a
value of a, and the weights of the previous observations decrease in reverse time. In moving average,
the average weight of the last s observations is assumed to be the same and equal 1‚𝑠. The weights of
all early observations are equal to 0.
In [14], the concept of average "age" in moving average was introduced in the form:
% -(-+%) %
𝑝 = - (0 + 1 + 2 + ⋯ + 𝑘 − 1) = !-
= ! (𝑠 − 1). (29)
As follows from (29), the average "age" of observations is the average of the "ages" of all individual
observations, taken with weights equal to the weights of these individual observations. With exponential
l
smoothing, the weight of an observation made at time k-l, will be ab , so the average “age” of
observations can be written as: 𝑝 = 0𝛼𝛽! + 1𝛼𝛽' + ⋯ + 𝑙𝛼𝛽' + ⋯ = 𝛼 ∑A #
-&! 𝑢𝛽 .
The condition of equality of the average "age" of observations in the moving average and with
exponential smoothing allows finding the relationship between the parameter a and the window s in
the form:
92'
4 '2E 92'
𝛼=
*
E
= E
= *
or [ 92' (30)
𝛽 = 95'
Of interest to us is the response of an exponential smoothing identifier to jumps in the controlled
signal. Let us assume that a unit jump occurs at time ka, i.e. bia = bi +1 for k ³ k. Using the standard
F
discrete z-transformation [19], taking into account that 𝜉[1] = , we can write:
F2'
EF F 𝛼𝑧 . 𝛽𝑇𝑜 /. / .
𝜉[𝑏% (𝑘)] = F24!0 . F2' or 𝑧−𝛽𝑇𝑜
. .+% = 𝛽𝑇𝑜 −1 . .+𝛽𝑇𝑜 + %+𝛽𝑇𝑜 . .+%, (31)
where T0 – monitored signal quantization period.
Going back to the time domain, we get
𝛼 4 !0 𝑘
𝛽% (𝑘) = 1 1−4!0 − 𝛼 1−4!0 𝛽 , (32)
whence it follows that an identifier with exponential smoothing comes to a new steady state bi + 1 or
faster than less 𝛽 (more a).
The general procedure for exponential smoothing (28) is intended for processing one-dimensional
signals xi(k). Within the framework of the problem we are considering, it is advisable to introduce
exponential smoothing of multidimensional sequences in the form:
s(k) = AX (k) + (I - A)s(k -1), (33)
where s(k) = (s1(k), s2(k) ,..., sn(k))T ,
A = diag (a1, a2, ..., an) – (n ´ n) - diagonal matrix,
I – (n ´ n) - identity matrix.
If you use the Trigga-Licch tracking signal to control changes
) ` (,)
⎧ 𝑇%)B (𝑘) = C% (,) ;
%
` ` ` ` (34)
⎨ 𝑇% (𝑘) = 𝛼% 𝑒% (𝑘) + 𝛽% 𝑇% (𝑘 − 1);
⎩𝑑% (𝑘) = 𝛼% |𝑒% (𝑘)| + 𝛽% 𝑑% (𝑘 − 1),
by analogy with (33), it is easy to introduce its vector analogue
) ` (,)
⎧ 𝑇 )B (𝑘) = ;
C% (,)
𝑇 ` (𝑘) = 𝐴` 𝑒% (𝑘) + p𝐼 − 𝐴 q𝑇 ` (𝑘 − 1);
` (35)
⎨
⎩𝑑% (𝑘 ) = 𝑑𝑖𝑎𝑔p𝛼% |𝑒% (𝑘 )| + (1 − 𝛼% )𝑑% (𝑘 − 1)q,
in this case, naturally, each component of the (nx1) -vector TTL(k) is controlled.
The occurrence of changes in the process is detected by testing for inequality 𝑚𝑎𝑥% (𝑇 )B (𝑘)𝑇 )B (𝑘 − 1) on
type (21).
To control variances, we introduce a vector of squares of current X(k) = (e1 (k), e2 (k) ,..., en (k))T
2 2 2
and an exponentially smoothed variance vector
𝑆8$ (𝑘) = 𝐴X(k)+(I-A)𝑆8$ (𝑘 − 1) (36)
It is clear that for a stationary signal and 𝛼% = 1‚𝑘 expression (36) describes the variances s bi of
2
estimates of the components bi, however, in a nonstationary situation at 0 £ ai £ 1, the growth of the
components of vector (36) indicates the occurrence of changes. Since in this situation the use of a
tracking signal is impossible, it is necessary to control the condition
𝑚𝑎𝑥% (𝑆8$ (𝑘) − 𝑆8$ (𝑘 − 1)) ≥ 𝛾8$ (37)
Analysis of the internal structure of a stationary multidimensional time sequence can be carried out
using its correlation matrix of the form:
%
𝑅 (𝑘, 𝜏) = " ∑"#$%(𝑥(𝑢) − 𝑥̅ )(𝑥(𝑢 − 𝜏) − 𝑥̅ )4 , 𝜏 = 0, 1, 2, . . . , 𝜏567 , (38)
which contains information about the autocorrelation and cross-correlation functions of all components
xi(k).
To detect changes in properties, you can enter an exponentially smoothed correlation matrix
SR(k,t) = a(x(k) - S(k))(x(k - t) - S(k))T , (39)
control by means of inequality
Sp(SR(k , t) 𝑆L) (k , t)) - Sp(SR(k - 1, t) 𝑆L) (k - 1, t)) ³ gR , (40)
where Sp(o) – matrix trace,
Sp(SR 𝑆L) ) – square of the spherical norm of a matrix SR .
Thus, based on the exponential smoothing methodology, it is possible to provide real-time control
over the changes in all characteristics of multivariate time series.
5.3. Detecting property changes in multidimensional time series based
on principal component analysis
An important problem in the analysis of large arrays (both in volume and in dimension) of
observations given in the form of time series is the task of compressing them in order to isolate latent
factors that determine the internal structure of the controlled signal, what in the end pursues the goal of
making the initial time series simplier interpreted from the point of view of detecting property changes.
It is worth using a well-proven factor analysis apparatus for these purposes [20], which is
characterized by using within which the method of principal components is most widely used (Karunen-
Loev transformation).
The analysis starts with the k ´ n observation matrix
(41)
formed by an array of k n-dimensional observation vectors x(u) = (x1(u), x2(u),..., xn(u))T, its
correlation (n ´ n) matrix of the form
' '
𝑅(𝑘) = , ∑,-&'(𝑥(𝑢) − 𝑥̅ )(𝑥(𝑢) − 𝑥’ )) = , ∑,-&' 𝑥 M (𝑢)𝑥 M) (𝑢), (42)
c
and x (u)=x(u)-𝑥 – mean-centered raw data.
The principal component method consists in projecting the observed input data from the initial n-
dimensional space into the m-dimensional (n>m³1) output space and reduces to finding a system
w1,w2 ,...,wm of orthogonal eigenvectors of the matrix R(k) such that 𝑤 ' = (𝑤',' 𝑤*' , … , 𝑤$' ) )
corresponds to the largest eigenvalue l, of the matrix R(k), 𝑤 * = (𝑤',* 𝑤** , … , 𝑤$* ) ) , the second largest
eigenvalue l and so on. Or the search for a solution to the matrix equation (R(k)-llI)wl = 0, such that
l1 ³ l2 ³ ... ³ ln ³ 0 и ||wl||2 = 1.
If we use the terminology of algebra, the solution to this task is closely connected with the problem
of finding the eigenvalues and rank correlation matrix. For geometry, the solution to this task provides
for a transition to a space of a lower dimension with minimal information loss. Finding the set of
orthonormal vectors in the input space that take on the maximum data variation is a task in the statistical
sense. In turn meaning of algorithmic solution to this task consists in the sequential determination
(selection) of a set of eigenvectors w1,w2,...,wm by optimizing each of the local functional, forming a
global test
'
𝐼N (𝑘) = , ∑O ,
#&' ∑-&'(𝑥
M)
(𝑢)𝑤 # )* (43)
with restrictions
𝑤 P) 𝑤 3 = 0, 𝑎𝑡 𝑙 ≠ 𝑝;
I (44)
𝑤 P) 𝑤 3 = 1.
Maximization of the local criterion is the way by which can be found the first main component. This
is what carries the maximum information about the monitored signal
%
𝐼89 (𝑘) = " ∑"#$%(𝑥 : (𝑢)𝑤 % )! (45)
applying Lagrange's standard method of indefinite factors [5].
The next step is to subtract from each vector its projection onto the first main component. component
xc(u). Then the first main component of the residuals is calculated, which is simultaneously the second
main component of the original data and is orthonormal to the first one.
After that, the third main component is calculated. Each source vector is projected onto the first and
the second main components. This projection is subtracted from each xc(u) and the first main
component of the resulting residuals is found, which is the third component of the original data. The
rest of the main components are calculated recursively according to the proposed and described
procedure.
It can be argued, that there is already a well-developed mathematical toolkit and software to
implement the Karunen-Loev transformation. But it should also be noted their drawback, which
concerns the need for a priori specification of the matrix X of a fixed dimension. But, when data is
received sequentially in real time, standard factor analysis procedures become inoperable.
Taking into account the above, for finding the eigenvectors of matrix R(k), it seems promising to
use recurrent online procedures, with sequential processing of observations of a multidimensional time
series x(1), x(2), ..., x(k), x(k +1)... and not to calculate the correlation matrix itself.
An artificial neuron based on an adaptive linear associator is described in [21] for calculating the
first main component in real time. In Figure 2, a diagram of this neuron is produced, modified to solve
the problem of detecting changes in properties in a multidimensional signal based on the analysis of the
main component. The learning algorithm for pre-centered data can be written as
𝑤 # (𝑘 + 1) = 𝑤 # (𝑘) + 𝜂(𝑘 + 1) ]𝑥(𝑘 + 1) − 𝑦(𝑘 )𝑤 # (𝑘)^ 𝑦(𝑘 + 1),
U (46)
𝑦(𝑘 + 1) = 𝑥 ) (𝑘 + 1)𝑤 # (𝑘), 𝑤 # (0) ≠ 0, 𝑦 ` (1) = 𝑥 ) (1)𝑤 # (0),
where h(k +1) is a tuning step parameter chosen small enough to ensure stable operation of the
algorithm [5].
Algorithm (46) provides the normalization of the vector w1(k): ||w1(k)||2 = 1, the vector w1(k ) itself
is an eigenvector of the matrix R(k). The maximum eigenvalue corresponds to it. The maximum possible
dispersion is a characteristic of the output signal y(k), which is explained by the content of the maximum
information about the multidimensional input signal x(k).
Then the output signal x(k) is exponential smoothed, which filters out the noise components x(k).
Changes are detected using a one-dimensional tracking signal 𝑇%)B (k) (24).
Figure 2: Modified neuron to detect changes in the properties of the main component of a multivariate
time series [22]
5. Conclusions
The tasks of registration, accumulation, processing, analysis, storage, search, and interpretation of
video streams are closely related to the analysis of time series, the main characteristic of which is
significant uncertainty. Methods for identifying time series outliers are the basis for almost any video
processing, search for homogeneous (in the broad sense, more precisely, in the sense defined by the
subject area) sequences of images.
An effective alternative to statistical methods that require the restoration and estimation of the
characteristics of the analysed data are adaptive procedures that allow solving problems of detecting
possible changes in real time under conditions of a significant shortage of a priori information. In
problems of video segmentation in the form of multidimensional non-stationary time series with a priori
unknown characteristics, it seems most appropriate to use the adaptive approach.
Three approaches to detecting changes in the properties of multidimensional time series induced by
video analysis in a certain feature space were considered.
Three approaches to detecting variations of properties of multidimensional time series, which were
induced by video analysis in a certain feature space were considered.
The first of the considered approaches is based on the use of custom models and is the most
reasonable from a mathematical point of view. However, to build an adequate mathematical model
capable of effectively detecting the emerging disruptions, large training samples may be required, which
is far from always possible, for example, with frequent changes in short-term plots.
The second approach is based on the use of multivariate exponential smoothing and is the simplest
from a computational point of view. At the same time, for its work, it does not require significant
number of information and allows you to effectively detect the emerging disruptions in the form of
sharp jumps in feature descriptions. At the same time, it is characterized by some inertia, and this leads
to a delay in the detection process, which, however, can be compensated for by using procedures for
regulating the smoothing parameter using a tracking signal, but, unfortunately, this will complicate the
computational model.
Finally, the third approach to detecting changes in the properties of multidimensional time series
using principal component analysis (Karunen-Loev transformation) makes it possible to use the entire
arsenal of existing methods for detecting mismatches of one-dimensional signals. However, its
application is complicated by the fact that the algorithm for training the neuron-compressor is not an
optimal procedure in terms of speed, which in some cases will lead to a "delay" in the learning process.
Thus, in conditions of a priori uncertainty (lack of completeness of description) when segmentation
of video streams, it is advisable to use all the proposed approaches in parallel. The approaches of
segmentation and video data processing proposed in the article can be introduced into the activities of
law enforcement agencies, since create good prerequisites for improving the performance of various
search engines, especially those related to searching through data content.
Analysis of the state and trends in the development of methods for searching for visual information
with query `ad exemplum` allows us to assert, despite numerous studies in this direction, the growth of
the accumulated video data and the intensification of their use require the creation of new high-speed
valid search tools.
The authors are specialists in the field of law, economics and IT, made an attempt to join forces to
solve the problem of introducing digital technologies into the criminal process.The problem is
extremely urgent today. The justice sector in general, and criminal justice in particular, requires
modernization as a reaction to changing reality.The emergence of new technologies that can increase
the efficiency of law enforcement, ensure the protection of individuals, society and the state, as well as
the coronavirus pandemic, which has caused the need to limit physical contact between people, are
factors that contribute to the search for new methods and tools for solving professional problems.
6. References
[1] Kryminalnyi protsecualnyi kodeks Ukrainy: Zakon Ukrainy № 4651-VI vid 13 кvitnia 2012 h.
URL: https://zakon.rada.gov.ua/laws/ show/4651-17#Text [Criminal Procedure Code of Ukraine
(2012)]
[2] Resolution 1604 (2008) Video surveillance of public areas. URL: https://pace.coe.int/
en/files/17633/html
[3] M.V. Zavalnyi (2020). Shchodo zaprovadzhennia oboviazkovoi reiestratsii kamer
videosposterezhennia, shcho spriamovani na publichni mistsia, ta poriadku otrymannia z nych
informatsii URL: https://library.ppss.pro/index.php/ndippsn_20200221/article/view/zavalnyi
[M.V. Zavalnyi (2020). Regarding the introduction of mandatory registration of video surveillance
cameras aimed at public places and the procedure for obtaining information from them].
[4] Lobach D. V., Mamychev A. Y., Miroshnichenko O. I., Moskvych L. (2021). Compreensão
doutrinária do terrorismo cibernético no contexto do intenso desenvolvimento das tecnologias de
informação e comunicação no mundo moderno. Laplage Em Revista, 7(2), p.201-208.
https://doi.org/10.24115/S2446-6220202172702p.201-208
[5] Mashtalir S., Mashtalir V. (2020) "Spatio-Temporal Video Segmentation.". In: Mashtalir V.,
Ruban I., Levashenko V. (eds) Advances in Spatio-Temporal Segmentation of Visual Data,.
Studies in Computational Intelligence, vol 876. Springer, Cham, First Online: 17 December 2019,
Part of the Studies in Computational Intelligence book series (SCI, volume 876), pp 161-
210, https://doi.org/10.1007/978-3-030-35480-0_4
[6] L. Liung (1991). Identifikatsiia sistem. Teoriia dlia polzovatelia. M.: Nauka, 1991. 432 s. [Lennart
Ljung (1991). System Identification. Theory for the User].
[7] Kaczmarz S. (1993). Approximate solution of system of linear equations. International Journal of
Control. 1993. 57. No 5. P. 1269-1271.
[8] Ye.V. Bodianskii (2000). Adaptyvne vyiavlennia rozladnan v obiektakh keruvannia za
dopomohoiu shtuchnykh neironnykh merezh. Dnipropetrovsk: Systemni technologii, 2000. 140 s.
[E.V. Bodyansky (2000). Adaptive detection of disorders in control objects using artificial neural
networks].
[9] S.V. Mashtalir, K.S. Shcherbinin (2010). Oblasti dostatochnosti i neobkhodimosti pri reshenii
zadach alternativnogo poiska videodannykh. Prikladnaia radioelektronika. 2010. T. 9, # 4. S. 580-
583. [S.V. Mashtalir, K.S. Shcherbinin (2010). Areas of sufficiency and necessity in solving
problems of alternative search for video data].
[10] A.P. Mikhieiev (1983). Iterativnaia filtratsiia s reguliruiemym vzveshennym nakopleniiem
informatsii. Dinamicheskiie sistemy:upravleniie, adaptatsiia i optimizatsiia. Gorkii, 1983. S. 94-
106. [A.P. Mikheev (1983). Iterative filtering with adjustable weighted accumulation of
information].
[11] S.V. Shilman (1983). Iterativnoie lineinoie otsenivaniie s reguliruiemym obiemom predistorii.
Avtomatika i telemechanika. 1983. # 5. S. 93-98. [ S.V. Shilman (1983). Iterative linear estimation
with adjustable prehistory volume].
[12] E.Y. Chow (1980). Issues in the development of a general design algorithm for reli- able failure
detection. Proc. 19-th IEEE Conference on Decision and Control. Albuquerque, 1980. P. 1006-
1012.
[13] K.D. Liuis (1986). Metody prognozirovaniia ekonomichieskikh pokazatieliei. M.: Finansy i
statistika, 1986. 133 s. [ Lewis, Colin David (1986). Industrial and business forecasting
methods].
[14] Yu.V. Chuiev (1975). Prognozirovaniie kolichestbennykh kharakteristik protsesov. M.: Sov.
Radio, 1975. 400 s. [Yu.V. Chuiev (1975). Predicting the quantitative characteristics of processes].
[15] D.C. Montgomery (1990). Forecasting and time series analysis / D.C. Montgomery, I.A. Johnson,
J.S. Yardiner. N.Y.: McGraw-Hill, 1990. 394 p.
[16] R.G. Brown (1963). Smoothing, forecasting, and prediction of discrete time series. N.Y.: Prentice
Hall Inc., 1963. 468 p.
[17] D.W. Trigg (1967). Exponential smoothing with an adaptive response rate. Operational Research.
1967. Vol. 18, No 1. P. 53- 59.
[18] S.V. Mashtalir, K.S. Shcherbinin, S.V. Postulga (2010). Analiz rezultatov segmentatsii
videodannykh. Intellektualnnyie sistemy priniatiia reshenii i problem vychislitelnogo intellekta.
(ISDMCI’2009): sb. nauchnykh trudov, Yevpatoriia, 17-21 maia 2010 g. Khierson: KhNTU, 2010.
Tom 2. S. 431-435. [S.V. Mashtalir, K.S. Shcherbinin, S.V. Postulga (2010). Analysis of the
results of the segmentation of video data].
[19] R. Izerman (1984). Tsifrovyie sistemy upravlieniia. M.: Mir, 1984. 541 s. [Isermann R. (1984).
Digital control systems].
[20] K. Iberla (1980). Faktornyi analiz. M.: Statistika, 1980. 398 s. [Iberla, K. (1980). Factor analysis].
[21] Cichocki A. (1993). Neural networks for optimization and signal processing. Stuttgart: Teubner,
1993. 526 p.
[22] S.V. Mashtalir, K.S. Shcherbinin, S.V. Postulga (2010). Poisk izmienieniia stsen v videodannych
na baze analiza rezultatov segmentatsii. Bionika intellekta. 2010. # 1. S. 65-69. [S.V. Mashtalir,
K.S. Shcherbinin, S.V. Postulga (2010). Search for scene changes in video data based on the
analysis of segmentation results].