Video surveillance systems, dash cameras and security systems have become an inescapable part of the most institutions ground environment. Their main purpose is to prevent incidents and to analyze the situation in case of extemporaneous events. Though as often as not it is necessary to increase an image segment many times over to investigate some incidents. Sometimes it is dozens of times. However, the obtained material is mostly of poor quality. This is connected either with noise or resolution characteristics, including focal distance. The paper considers an approach for improving image segments, which were obtained after multiple zooming. The main idea of the proposed solution is to use methods of blind deconvolution. In this case, the selection of restoration parameters is carried out using evolutionary algorithms with automatic evaluation of the result. That seems like the most important detail here is pre-processing besides noise minimization within the image, because when the image is repeatedly enlarged the effect of the noise component also increases. To avoid this thing, we suggest using ordinal statistics and average convolution for a series of images. The proposed solution was implemented as a software product, and its operation was tested on a number of video segments made under different shooting conditions. The results are presented at the end of this article.
High-quality image improvement at multiple zooming (super-resolution, SR) has
always been and still remains an actual and popular area of mathematical and algorithmic
software development. The popularity of these algorithms is due to the fact that no
matter how large the resolution of modern cameras is, it is still insufficient and will
always be lacking [
One of the multiple zooming problems is the distortion of the image segment, its susceptibility to a large amount of noise, blooming and blurring, which is not visually perceived at all and is often not suitable for analysis without additional processing. Therefore, it is very important to return as much information as possible and restore the image.
Currently, there are three full-fledged directions of healthy development in the field
of super-resolution, which differ in the principles of operation and master data, entering
the system input. The first direction is simple image zooming with post-processing of
the resulted image. A single frame is used as the source images. At the same time in
case of simple zooming by means of the nearest neighbor method the image will look
like a mosaic made up of squares when being zoomed repeatedly, which is bad enough.
Therefore, the approaches in this category are aimed at removing the mosaic effect,
making the image more readable and beautiful without adding new details to it. The
most well-known representative of this class of algorithms is interpolation-based
approaches (for example, the bicubic interpolation method [
The third class of algorithms is aimed at working with a series of images. In this case, it is assumed that there is not just a separate image, but a video stream, a series of snapshots of the same object. This allows trying to combine data from different images, extract different details that are possessed by one image but be lacking in another frame and also reducing the impact of noise and other distortions. A number of retrain positive frames provides more opportunities, but also requires the use of more complex approaches and algorithms. This work is devoted to the development of the third group of methods. It offers an approach of truncated convolution of a series of images taking into account ordinal statistics, which allows reducing both: the influence of the noise component and distortions in the averaging process, thereby increasing the influence of
Method for Improving an Image Segment in a Video Stream Using Median Filtering… 3 the sections having maximum information content. After that, we suggest using the method of blind deconvolution based on evolution algorithms to diminish defocusing. The scheme of this option is shown in Fig. 1.
As can be seen from the scheme presented above, the proposed algorithm has two stages of interest - reducing the influence of noise using information from a series of images and blind deconvolution based on evolutionary algorithms. Let's look at these steps to some detail. 3
Multiple zooming makes you solve a number of problems. And one of them is the influence of the noise component. Any image received from a camera or a still-shot camera has interference and noise. Noise may not be visible to the naked eye at normal resolution, but it can be highly visual if enlarged. This is due to the fact that when we make zooming, we increase the contribution of noise to the result owing to the expansion in the amount of it.
Noise also greatly affects the result if the restoration process is used further (for example, blind deconvolution based on the Wiener filter). Most restoration methods are very sensitive to noise and can result in the wrong way. If you have a whole series of synchronized images (a video stream) instead of a single snapshot, you can use ordinal statistics and a truncated average as preprocessing. The median filter was considered as an alternative, but it showed a slightly worse result in the presence of combined noise. You can use a winsorized mean filter for these purposes also.
In ideal shooting conditions, the following condition must be met: the value of any pixel in the first image in the position [i, j] must completely match that of all other images in the series: (1) (2) (3) (4) ∀ , , : 1[i, j] == [i, j].
However, this is not the case in practice. A number of images have a noise component. We will assume that the noise is a random variable (in most cases, it is). In this case, the Imgk[i,j] pixel will have a real value in a number of images, and a distorted value for some of them. In order to reduce the effect of distortion, you can find the arithmetic mean of each pixel within the entire series consisting of n images (we choosen n equal to 20 in our experiments):
[i, j] = ∑ =1 [i,j] .
The averaging results in the noise component as if got smeared across the entire series of images, thereby reducing the contribution to each specific image. However, experiments have shown that it is better to use a truncated average instead of simple averaging. That is why first it is critical to sort the values of each specific pixel by brightness value. Let's assume that the interference caused either a slight increase in brightness at this point or inversely, its decrease. So it is necessary to discard the first and last m statistics, and average the remaining part:
Let's look at this approach in more detail. Let's assume that there is an array of images of the same object: its shooting conditions did not change during the entire series: { 1, 2, … ,
}. .
As shown by experiments on real images, this preprocessing gives more accurate results during further restoration and an increase in the convergence rate of the restoration process by approximately 10%, when based on the evolutionary algorithms proposed and described later in this article. 4
The next step, after receiving a snapshot with a reduced impact of the noise component, is the restoration stage, shown on the right in the General diagram (Fig. 1). As proved by numerous experiments, after multiple zooming, the segment of interest is almost always blurred and poorly readable. This is comparable to an image taken with the wrong focal length. Therefore, the logical step here is the attempt to use methods for restoring distorted images based on inverse filtering.
Conventional methods of inverse filtering allow restoring the image quite well. But these methods have one great disadvantage – it is inevitable to give a detailed description of the distortion model, including all its parameters, such as blur radius and noise.
Method for Improving an Image Segment in a Video Stream Using Median Filtering… 5 If you choose one of the parameters incorrectly, the restoration result will be worse than even the original image, and the restoration will nothing but spoil the image. With multiple zooming, it is not possible to predict the distortion parameters and build a fullfledged model of it.
To solve this problem, methods of blind reverse convolution (blind deconvolution) are actively used. The purpose of this approach is to find an unknown distortion function and its parameters. To speed up selection, we suggest using genetic algorithms, and to evaluate the result at the stage of choosing alternatives, we use automatic metrics based on gradient and maximum likelihood estimates.
The use of genetic algorithms cannot be attributed to traditional approaches for
image processing, but in some cases they work giving not bad results [
However, later, in the course of conducting experiments, it was observed that the choice of the initial options for the blur radius made formalized dependency on the zoom scale more difficult. The smaller the zooming, the lower the maximum blur value can be selected in the original population. But in some cases, when the original image is multiply zoomed, 10% is not enough. However, this fact is not yet fully developed and requires further research and analysis in the future.
Crossing and mutations. The main stages in the work of the genetic algorithm are crossing and mutation. When crossing different biological units, the binary exchange of parts of information encoding the radius of blurring occurs. In this way, the model parameters are modified and a new population is formed for evaluation.
Right after crossing, a number of exponents undergo mutation. The paper identifies two types of mutations: weak and strong. In the case of weak mutations, the blur radius changes to a random value of a small nominal value (up to 3, the value was chosen a posteriory for increasing the convergence rate). Such mutations lead to the modification of units and allow expanding beyond the original population in the case when the population includes samples with very similar blurring radii. Areas with mutations are also added to the resulting population.
The second type of mutation (strong mutations) is performed with a significant change in the blur radius. For this type of mutation, only a few biological units are selected and the blur radius in them changes by a random amount up to 5% of the image size. This stage allows exiting of the local maxima areas. However, a large number of such mutations just contributes to the delays of the convergence process. Therefore, the number of such mutations was selected in the region of 2 per iterate.
Evaluating function of the image restoration quality. After obtaining the population (variants of the distortion model), they must be evaluated using the goal function to select the most adapted samples (images with good restoration quality). The goal function was chosen to reverse restoration using a Wiener filter and to estimate the average gradient value of the resulting image. The gradient proved to be quite good when restoring defocused images. The Sobel operator was used to calculate the gradient.
The choice of gradient as an evaluation function was made for the following reasons.
Research in the field of image processing and analysis shows that despite the variety of
shapes of the images themselves, their gradient histogram is of similar appearance
[
This estimation approach is very popular among blind deconvolution methods. However, in the course of experiments, it was noticed that it is not suitable for images with a large number of sharp changes (for example, the image of text on a white background). In addition, it has one more disadvantage: for evaluation of the image quality the etalon gradient histogram is required to be compared with, which is not always convenient. In real conditions, it is not always possible to select such a histogram in advance, and the use of some average analogue often does not exceed the use of a simple value of the average gradient over the entire image. Therefore alternatively in our system, we used a slightly different approach, which also uses the gradient value, but focuses on the average value across the entire image. In this case, you do not need to set the etalon gradient histogram, and the system adapts itself to the current frame and the data type in it.
Additional restrictions of the goal function. Using the average gradient generally gives very good results. However, sometimes there are situations in which a high gradient value does not produce better results. This can happen with images having a large number of small objects (Fig. 3a), which turn into dots when blurred, thereby increasing the average gradient value (an example of this case is shown in figure 3b).
To eliminate the defect additional restrictions when evaluating the quality of alternatives were set. The MSE (Mean Square Error) metric was chosen as the restriction, which is often used to evaluate the similarity of the restored image with the distorted one. A large value of this metric shows a significant difference between images. Using this metric, as can be seen the images in Fig. 3a and 3b are within only 6 conventional units (which is inappreciable). So these alternatives should not be considered and a number of them should be excluded from the resulting sample (but not all of them, in order not to exclude absolutely the entire population providing that the restored image really should be similar to the original one).
Optimization completion criteria. As a completion criteria of restoration stage a number of populations was chosen, which went wrongly in regard to the restoration result. Ten populations were selected acting as this value. After completing the iterative algorithm, the value of one of the biological units is selected as parameters of the distortion model and used to restore the image. The result is shown in figure 3c.
Nevertheless, as experiments have shown, it is not always necessary to choose a sample with the best indicator of the goal function. This is due to the fact that the average gradient estimate does not always give an optimal result if the population units are close in value to the goal function. Therefore, the following algorithm was proposed to select the final version of the distortion model. 1. Select two best samples from the point of view of the goal function. 2. If the value of the goal function of the first sample significantly exceeds the value of the second one (the difference in the values of the average gradient is more than 15), then the first is selected as the most suitable. This is a situation where there is an obvious favorite and it has not changed for a long time (more than 10 iterations). 3. If the value of the goal function of the first and second units do not differ much (the difference is less than 15, samples are similar to each other), then the structure similarity metric (SSIM) is calculated for the restored images to compare with the original one. As a result of comparison, the image with less SSIM metric value is selected (since it is necessary to select an image resembling a distorted version to a lesser extent).
Applying another metric (SSIM instead of MSE used in the previous stage) showed a better result. The SSIM metric did not perform in the best way (although it was quite good) at the intermediate stage of evaluation, while at the final stage it allowed achieving a qualitative result. 5
For experimental study a utility was developed for processing data from a video stream with the ability to improve separate segments. The utility was written in C++ using the OpenCV library.
A number of videos of the sights of the city of Bryansk were taken as source data. Shooting was done on a camera with a tripod. This is due to the fact that this work does not concern issues related to the mapping and synchronization of frames during processing. Frame synchronization is a feature length topic that requires separate study. In addition, shooting from a tripod is the most approximate for the conditions of use in surveillance cameras, in which the camera itself is fixed rigidly.
A series of experiments was performed, the purpose of which was to search for and identify the strengths and weaknesses of the proposed method, determine the limits of its applicability, and confirm its performance.
The first series of experiments was aimed at determining the effectiveness of the first stage of the system – filtering a series of images based on ordinal statistics and average convolution. An example of this is shown in figure 4.
1
The figure gives an example of one of the video stream frames, which shows the Bryansk regional Puppet Theater. With sufficient zooming of the image segment, being highlighted by the marker, you can see some noise that hamper perception and obstruct the view (enlarged segment is shown on the left). But a detailed analysis of a series of images proves that these interferences are random. Therefore, the proposed version of preprocessing based on ordinal statistics and average convolution really produces a successful result (the segment after processing is shown on the right).
However, it is worth noting some minor effect of the proposed filter. The image becomes slightly out of focus and a number of objects will increase slightly in linear dimensions. This effect is most likely due to incomplete synchronization of images during averaging. This side effect can be eliminated by supplying an additional synchronization stage to the filter based on calculating key points. Though, at the moment we have not conducted such studies yet and this is the direction of further development of the work.
The second series of experiments was to test the efficiency of the distortion restoration stage based on blind deconvolution using a genetic algorithm. An example of restoring of two image segments is shown in figure.
2
As an example of restoration, we selected two tablets in the image of the Bryansk regional Puppet Theater, blurred after enlargement due to the fact that they fell out of the camera's focal length (the enlarged segments are shown in the figure on the left). After the blind deconvolution stage, this defocus was significantly reduced and the outlines of the letterings became clearer (the result is shown in the figure on the right).
Nevertheless, it is worth making a number of observations. First, like any method of blind deconvolution, this approach can lead to side defects (for example, ripples or false contours). This effect is not always apparent and in some cases is almost invisible and does not affect further analysis. But if such behavior is undesirable, it can be avoided by making parameter adjustments at the last stage.
Secondly, the described method cannot be considered a panacea in absolutely all cases. As shown by the research, it does not give bad results if zoomed and processed image segment was not in the focus of the lens (which is a fairly common situation when analyzing frames from video surveillance systems that require additional processing). But it does not enable you to restore image details if there is no information about them at all in the video stream (for example, when increased tenfold, if the analyzed segment is up to several pixels in the original image). The paper presents a method for improving image segments obtained from a video stream. The main features of this method are using of truncated average convolution at the preprocessing stage and applying of blind deconvolution based on genetic algorithms. Research has shown that in some cases this approach is quite effective, especially in a situation where the enlarged image segment is not in the focal length of the camera. However, the proposed solution can be developed more in the direction of further studies.