The study proposes a deep neural network architecture to monitor the dynamics for a state of a complex technological object according to the data received in the form of images. The paper also contains recommendations for the architecture adaptation to a specific application. The developed architecture is based on the cascade use of convolutional neural networks for processing multi-channel video information from different technological zones of one object.
Now deep neural networks (DNN) represent the most practically significant direction in the development of artificial intelligence methods. DNN are actively used in the systems of video analytics to obtain metadata from a video stream. From the technical point of view, video analytics is a software and a hardware complex for the intellectual analysis of the events that fall into the sector of video surveillance systems and undergo deep processing by software tools. On average, only 10% of data, which a camera is able to give for processing, are used. The intellectualization of this process allows increasing the part of the useful information implementation.
Initially, in the systems of industrial safety video analytics was applied as a
detection of an object movement or crossing of a control line, the objects identification
(people, transport, luggage), their behavior estimation [
– search and prediction of equipment failures to reduce the probability of incident situations;
– implementation of multidimensional machine vision based on processing information from a large number of sensors in order to monitor technological processes (TP) in real time and predict their behavior;
– business analytics based on the generation of metadata for a state of a production process by intellectual cameras, as well as the detection of hidden regularities in visualized information about the results of an enterprise commercial activity.
Most of the noted directions for the artificial intelligence methods implementation
consider video data processing and include the estimation for the change rate state of
the object under observation. The image changes of the object under study can be
applied as a visual prompt in decision–making process as the meaning of many
actions is precisely in the dynamics, it is enough to observe the movement of individual
points in order to recognize the event. In this case the changes are understood as a
wide set of characteristics for images, i.e. the shifting of any objects and contour in
the background, changes the brightness of the elements, texture modification [
The problem of developing and adapting machine vision methods and algorithms to assess the rate for the change of TP state, taking into account the specificity of production, is actual due to the significant diversity of nature of the observed objects and processes. 1
Complex TP are characterized by the significant duration not only in time but in
space as well. This fact specifies the reasonability to include into APCS not only
signals from standard control and measuring equipment, but from the additionally
installed systems of visual control for technological zones with high responsibilities
as well [
Suppose there are kz of technological zones for which video cameras are installed.
Each video camera gives a stream of shots with the resolution of n[ip]×m[ip] pixels,
where ip=1, 2, …,kz and frequency f. As a result, video data from all the cameras are
mapped by tensor X of the sixth rank with the form: a camera, samples, shots, height,
width, color [
Different camera models allow forming a video with the frequency of shots in a big range; usually it is from 10 to 60 fps (frames per second). Then, to be processed by the convolutional neural network, the discretization interval at time Δt(i), i=1, 2, …, I, where I is a number of information video channels, for i-th information channel, is chosen to be more than one second. This choice is based on the assumption that the inertia of TP allows to do it without violation of Kotelnikov’s theory which sets the maximum value of the discretization interval, at this value the accurate restoration of initial continuous signal is possible. If the discretization interval is needed to be less than one second, then it is necessary to choose a video camera model more carefully, but the methodology of the given bellow approach to the dynamics recognition is not changed.
It is required to develop an architecture of a deep neural network to detect metadata
from the forming tensor X, the metadata provide the recognition and forecast for
“movement” (evolution) of TP in time. The estimation for the recognition quality is
implemented on the base of confusion matrix CM [
DNN find their application in algorithmic support of video analytics systems for
various application areas, for example: to detect technological defects [
However, in most cases these algorithms do not support simultaneous description
of an object by its image and motion, that will allow recognizing the events even at a
low resolution and predict the evolution of an observed object state. The exception in
this case is only a direction connected with the recognition of people actions [
One of the popular methods, called “sliding window”, consists of several stages: fragment selection (spatio-temporal parallelepiped); solution of the images classification problem and search for the objects for three-dimensional spatio-temporal volume. However, it does not suit the automated recognition of changes, as it requires prior indication of window borders on the image which makes it difficult to be applied when the boundaries of this window need to be shifted in the observed technological process.
In other methods for analyzing video sequences the basic tool is the concept of
optical flow. This approach was first proposed by Bruce D. Lucas and Takeo Kanade
in1981. Optical flow is often defined as a vector field or an image of objects apparent
motion, surfaces or scene edges resulting from the motion between an observer and a
scene [
The proposed algorithm is based on the application of DNN ensemble, the structure of which has a temporary time delay unit to enable the recognition of objects dynamics by video data.
Advances in the use of deep neural networks in computer vision systems for various purposes provide reasons for optimism in the case of recognition of TP processes dynamics. This approach reflects a modern paradigm of Software 2.0 which unlike Software 1.0 does not imply the explicit writing of an algorithm , it provides the creation of a neural network with a specified architecture which learns (adjust) itself to solve a specific applied task.
Deep neural network models hierarchical abstractions in data using architectures
which consist of cascading ensembles of nonlinear transformations (filters). Today
there are some popular DNN architectures: neocognitron, autocoder, convolutional
neural networks (CNN), Boltzman machine, deep trust networks, long short-term
memory networks, controlled recurrent networks, residual neural networks [
The architecture of a neural network defines the hypotheses space, i.e. the number
of classes for the input data sets splitting. The proposed architecture for a deep neural
network to monitor the object dynamics is in Fig.1. It uses the successful practices of
neural networks ensemble application [
After the calculation for the intervals of discretization according to time Δt(i) for all channels the minimum Δt = min i (Δt(i)) is chosen for further synchronization and unification of video data transformation performed by the neural network.
Also, to unify further transformation the input multichannel images with the resolution of n[ip]×m[ip]pixels, before being fed to the neural network input, are normalized to one dimension of n×m pixels which is smaller than the minimum from n[ip]×m[ip].
In the time delay block the shift is done by moment Δt of image receiving which makes it possible to calculate discrete analogues of derivatives when defining the values changes at neural networks outputs.
The input cascade of a deep neural network contains some CNN operating in parallel classifying images from cameras of a corresponding information channel taken at intervals Δt. This procedure consists in forming output channel vector V (j|i), j=1,2,…, cl(i), where cl(i) is a number of classes for i-th information cannel, at each moment Δt.
Time delay a t a d o e d i v e t a t s s s e c o r P
CNN of an input cascade forms the elements of vectors V(j|i), which take the values in the range from 0 to 1. This reflects the i-th channel CNN degree of confidence in membership of the parameter controllable according to the image to a particular class at moment t(k)= kΔt, where k is a sequence number of time discrete.
The time interval during the technological process is divided into fragments with ΔT duration. Consider the fragments for all information channels are equal and defined by the requirements for the periodicity of information flow to APCS and characteristics of the most frequency-critical channel.
By moment T(ζ)= ζ ΔT, ζ =1, 2, …,ψ, where ψ is a number of fragments with ΔT duration for each i-th channel, the matrix of classification results can be formed: where element V(j|i, k) means the j-th CNN output of the input cascade for the i-th information channel at time discrete kΔt.
Matrix (1), in fact, reflects CNN degree of confidence changes in classification results when TP passes interval ΔT under number ζ.
CNN internal cascade receives the tensor consisting from the combined particular matrixes (1) for all information channels. At the output of the internal cascade matrix MS is formed. It is an analogue of matrix (1), containing more number of elements according to the numbers of information channels and the classes of TP state for each channel.
To have an idea about the dynamics of the entire technological process tensor DV is calculated. Tensor DV contains the relations of MS matrix elements increments to Δt discretization interval. The range of this tensor is equal to three, and its ζ-th cut has a form of: ... ... ...
MSξ (1,I )−MSξ −1(1,i) ⎞
Δt MSξ (2,I )−MSξ −1(2,i) ⎟⎟ Δt ...
⎟⎟. ⎠ (2)
Cut (2) can be interpreted as an image fed to the output of CNN cascade. The sense load for the elements of cut (2) can be matched with the analogue of derivatives for the continuous functions as they reflect the confidence change of neural network in image membership to a certain class. The rate of change can be used to forecast TP development.
It should be mentioned, that the number of classes for different information channels and different technological zones can be different. Thus, to provide the proportionality when feeding tensor DV for neural network processing some cuts contain zeros on the places of redundant classes.
Tensor DV is formed on the base of the data about the CNN confidence changes in class membership of information channels parameters reflected in matrix (1). It allows assuming that hypotheses space formation can provide good forecasting results with the help of the proposed deep neural network architecture. 4
The recommended choice for the deep neural network architecture means to insure
the correspondence for the number of channels of the video information flow with
the number of convolutional neural networks in the input cascade. In this case the
authors give the results of simulation experiment when processing ingots images of
aluminum alloy with the aim to forecast the time from its complete melting. The
melting process takes approximately 300 seconds; the aggregate state is estimated by the
surface image observing through a viewing window fitted on the furnace door [
The model program was performed in IDE Spyder from Anaconda (version for
Linux) in Python 3.6. CNN were created with the special neural network library Keras
which is the add-on the framework of Tensor Flow calculations [
In the considered example only one informational channel from the technological melting zone is used, thus the structure of the applied network is significantly simplified. The network contains seven alternate convolution layers and subsamples and one output fully connected layer with four outputs according to the number of defying classes for the substance aggregate state: class «solid» (0 – 269 sec.), «initial transit » (270 – 279 sec.), «final transit» (280 – 289 sec.), «liquid» (290 – 300 sec.). Thus, the time interval is connected with the class, therefore, when making the classification the time of the aggregate state occurrence is forecasted.
Fig.2 shows the images of a melting zone taken in different moments, Fig.2b shows tensor DV image reflecting the dynamics of images presented in Fig.2a. The sequence of 2b images forms the dynamics trend for the melting process which is recognized by CNN.
The initial learning sampling for each class had 1600 examples; the testing sampling had 400 examples. In addition to the standard Dropout method, used in CNN, augmentation was implemented to reduce the possibility of the network relearning. Shifts, scale changes, rotation, mirror reflection, affine transformations were realized for the initial images of the ingots working surface. As a result of these procedures, the total size of the training sampling was 32,000 examples; the total size of testing sampling was 8000 examples. The network was trained during 110 epochs. Categorical entropy was set as a loss function for CNN.
The splitting of video data into shots with conversion into jpg format is performed with software utility Free Video to JPG Converter with a given discrecity of one second. This approach allows obtaining sufficient learning sample volume for CNN as a great number of images can be detected even from the video recording of one melting process.
The learning is conducted on video card GeForce GTX 1060 installed on Asus FX502VM notebook with CPU IntelCore i7-7700HQ, which provides more than twenty fold time gain comparing with the learning on a regular notebook processor. The quality of CNN learning is reflected in accuracy metric which is 77 % on testing samples. The graphics for loss function and accuracy function are shown in Fig. 3.
Сonfusion matrix (CM), used to estimate the classification quality, has four rows and four columns in this case. Its columns reflect the factual data; its rows show the results of the classifier work.
Fig 3. Graphics for behavior of loss function and accuracy function.
When filling the matrix the number on the cross of the class row, returned by the classifier, and the class column, to which the object really belongs to, increases by one. In this example every time interval corresponding to different melting stages is divided into 10 subintervals ΔT. After the conducted experiments matrix CM is filled: different information flows into the entire set and increase the level of the presented data abstraction. In the output cascade of a neural network the time delay in images processing is used to enable the object state recognition dynamics and its forecast.
2. Recommendations on the choice and adaptation of the neural network architecture depending on the number of technological zones and the number of information channels are given.
3. The study presents the results of simulating experiment which show the efficiency of the proposed neural network architecture. The results can indicate the reasonability of the architecture use in various application areas where it is necessary to control the dynamics of the processes by the available video information. 6
The reported study was funded by RFBR according to the research project № 19-01-00425.