-The paper provides examples of convolutional neural network architectures, the corresponding activation functions, and the organization of their interaction in the learning process. Networks interact with each other according to the architecture of generative-adversarial networks. For the task, the NEXET 2017 data set was filtered and formatted. Studies of the architecture of neural networks and varying the volume of the training sample to solve the problem of image styling were carried out.
NEXET 2017, loss function,
The use of artificial neural networks is relevant for a
wide variety of applications [
There are special algorithmsto solve the problem of
detecting objects in real time [
The image processing industry is one of the fastest
growing in the world. Film studios hire hundreds of
designers to meet the deadlines for creating CGI
(ComputerGenerated Imagery) videos. The work being done is
extremely painstaking and of the same type. These properties
characterize it as a potential candidate to replace it workers
with artificial neural networks (ANNs). Particularly complex
and monotonous is the change in the lighting conditions of
the images, in particular the transformation of the daytime
image into nighttime and vice versa. Until 2014, this problem
did not have a solution allowing stylization in a resolution
acceptable for the current generation of image formats (1280
× 720 or 1920 × 1080 pixels). In 2014, the architecture of the
Generative-Adversarial Networks (GAN) was invented [
In 2016, a work was published that proposed a new
architecture based on GAN — the cyclic
generativecompetitive network [
The research has shown the possibility of using the architecture of a cyclic generative-adversarial network to solve the problem of styling images for day and night lighting conditions. The relevance of this work is due to the need of the film industry to simplify the process of shooting video by replacing the post-processing of a number of effects recreated on stage. In particular, one of the most problematic effects is shooting in low light conditions or night shooting. Equipment for night shooting is expensive, which makes the need for styling daytime images into nighttime images high.
II. ARCHITECTURE AND LOSS FUNCTIONS FOR NEURAL
NETWORKS
According to the GAN architecture, two networks are involved in the learning process - the generator and the Layer type Conv Conv Conv Conv ResNet ResNet ResNet ResNet ResNet TrConv TrConv TrConv Conv
Out Layer type Conv Conv Conv Conv
where ( ) and ( ) are functions of discriminators Let’s introduce the cyclic error function for transition from one class to another: =
1 3× ×
∑ =1 ∑ =1 ∑3=1|
− ̃ |,
(1)
(2)
(3)
discriminator [
multilayer counterpart. This problem is called the degradation of the ANN. The addition of layers is possible due to the fact that the dimension of the output and input of the block must coincide. It is also optionally possible to add an activation function at the exit from the block. The transposed characteristics convolution applied
layer to the on the basis
of
input
reproduces
the
data
possessing these characteristics[
The general function of the cyclic loss, taking into account (3), will take the following form: = 1 ( 2 + )
The resulting loss functions of the generators, taking into account (4), take the following form: → = ( ( → ( )) − 1)2 + λ → = ( ( → ( )) − 1)2 + λ (4) (5) (6)
III. TRAINING DATA FOR THE NEURAL NETWORK For training the neural network, a part of the NEXET 2017
database photographs was selected.
This database consists of from auto-registrars in the resolution of 1280×720. The dataset contains 17000 images. To divide into day and night sets, 200 images were manually selected (100 for each class), and the brightness of each of them was calculated using the following brightness formula from the
= √0.299 × 2 + 0.587 × 2 + 0.114 × 2, (7) where r, g and b are the discrete values of the RGB components of the pixel in the range [0; 255]. Thereafter all the values were normalized by dividing by the maximum value. The results are shown in Figure 1. 11,442 day images were obtained, of which 500 random images per class were selected for the test sample. Due to the extremely limited resources for such tasks for neural networks, each image was cropped to a 1:1 aspect ratio and scaled to 256×256 pixels.
IV. CONDUCTING EXPERIMENTAL RESEARCH AND ANALYSIS
OF THE RESULTS CPU —intel core i5-2500, 3.3 GHz.
RAM—8Gb, 1333 MHz.
GPU —Nvidia GTX 1060, 6Gb GDDR5.
OS —Windows 10 x64. Ver. 1903.
A program was written that implements algorithms for training ANNs and styling images using the designed model using the following libraries:
A neural network has the following parameters: Learning rate —2×10−4 for all networks.
The size of the training set is 1000 elements. The size of the test sample is 100 elements. The number of eras is 100. A graph of the errors of discriminators and generators is shown in Figure 2.
Experiments have shown that the conversion from day to night does not always work correctly - only the brightness of some elements of the scene increases, while the sky does not stylize. Increase the number of training examples to 10,000 daytime and 4,500 nighttime. 500 images from each class are used as test cases. Due to the sharply increased data volume, we reduce the number of epochs to 20. The graph of the errors of discriminators and generators is shown in Figure 4.
The result of the conversion of test images is shown in Figure 5.
We will change the learning speed by several orders of magnitude in order to find out whether the learning speed standard for most tasks is = 2×10−4 acceptable for this task. The next experiment has been done on the same test image the result of processing neural networks trained on an incomplete (1000 images in each class) data set with different learning speed indicators: 2× 10−4 , 2× 10−3 and 2×10−2 . The number of epochs is 20. The result of the experiment is shown in Figure 6.
Comparing Figures 3 and 5, we can notice that the neural network obtained as a result of the first experiment either does not substitute the sign of light sources at night, or does not fully decode it, since the manifestations of this sign are noticeable in Figure 5, that indicates the need for a large number of eras for this styling task.
In this research, a software package was developed to demonstrate the operability of the cycleGAN architecture in image styling tasks. Training and test samples from the NEXET 2017 data set were generated.The studies with the designed software package have shown the possibility of using the architecture of a cyclic generative-adversarial network to solve the problem of styling images for day and night lighting conditions. The solution to this problem is relevant in the film industry for creating CGI-video.
In the course of the work, the following tasks were solved: the cycleGAN architecture was implemented, a database for training and testing was formed, the ANN was trained on a complete and incomplete set of training data.
Studies have shown that to solve the problem of styling images for day and night styles using ANNs, one should maximize the number of unique elements of a training sample. This allows you to reduce the result of the sum of loss functions by 25% with fewer eras, which indicates an improvement in the quality of stylization. Also shown that for this problem in the selected ANN configuration, the optimal learning rate is 2×10−4.