In this paper we propose exemplar-based 3D texture synthesis method which unlike existing neural network approaches preserve structural elements in texture. The proposed approach does this by accounting additional image properties which stand for the preservation of the structure with the help of a specially constructed error function used for training neural networks. Thanks to the proposed solution we can apply 2D texture to any 3D model (even without texture coordinates) by synthesizing high quality 3D texture and using local or world space position of surface instead 2D texture coordinates (fig. 1). Our solution is based on introducing 3 different error components in to the process of neural network fitting which helps to preserve desired properties of generated texture. The first component is for structuredness of the generated texture and the sample, the second component increases the diversity of the generated textures and the third one prevents abrupt transitions between individual pixels.
model
2021 Copyright for this paper by its authors.
(
tries to modify a random noise texture until property functions of generated and exemplar images become similar to the function of the properties of the exemplar texture. Either neural network based approaches and pyramidal synthesis can be attributed to that group [15]. As the pronounced properties of this group of methods, one can note high variability of the synthesized textures, low speed, as well as in many cases poor results when working with textures that have high resolution or complex structure.
have been eliminated in recent works by consistently doubling the resolution of the generated textures generate the new texture. Typical representatives of this group of methods are per-pixel [16] and perpatch synthesis [17, 18]. As a characteristic of this group, one can designate the high speed and acceptable results on texture sampling. The disadvantage of this group of methods is the low variability of the synthesized textures. 2.1.
network generators first define the image property function. For this, an exemplar of texture is fed into the neural network, and the activation function is calculated for each layer of the neural network. Each activation function generates a certain set of filtered images - the so-called feature maps. Each layer with filers will have same amount of feature maps, each of which has a total dimension . Therefore, all feature maps can be stored in matrix ∈ × , where – is an activation for j filter at spatial position inside layer .
After finding all feature maps, for each of them we can calculate the Gram matrix ∈ × : A set of Gram matrices { 1, 2. . . , } of neural network layers 1, 2, …, L is a possible way to describe the properties of an image. Then, to define the similarity function of the exemplar texture and the synthesized texture, we can use the following error : = ∑ .
1 ( , ) = ∑ ‖ ( ) − ( )‖ , 2 neural network to the error , ‖ ‖
– Frobenius norm. where и – generated texture and source texture respectively; – contribution of each layer of
Thus, to synthesize a new texture, we need to minimize the specified error between exemplar
texture and the random noise texture. This can be achieved by applying gradient descent to the noise
texture while calculating the gradients using backpropagation of errors.
(
[10-14]. The algorithm can be described as follows: we have a certain texture generator based on convolutional neural networks and which depends on a fixed set of parameters . Three-dimensional white noise of several different resolutions is fed to this generator, and the resolution of the synthesized texture will depend on the resolution of the supplied noise. Passing noise through itself, the generator will produce a three-dimensional texture of the specified resolution.
directions of the coordinate axes, and using some function that describes the properties of the image, it will calculate the error between the planes representing generated 3D texture and the exemplar textures we have. Setting the necessary parameters of the generator to generate a high-quality texture will be done using gradient descent and back propagation of the error similar to the two-dimensional case.
The generator architecture is shown in fig. 2. It is a sequence of convolutional blocks, upscaling blocks, and concatenation blocks: A convolution block is a sequence of 3 convolutional layers, the first two of which have 3x3x3 kernels, and the last - 1x1x1. This is followed by the batch normalization layer [20] and the Relu activation function [21]. Thus, after this layer, the size of the texture is reduced by 4. The upscale block doubles 3D texture resolution (each voxel will be copied 8 times). The concatenation block performs batch normalization and then concatenates our textures into channels. If the textures are Are different in size then they are cropped to a smallest size.
As already mentioned, to calculate the similarity between the three-dimensional texture obtained by the generator and the exemplar texture we have, the first one will be divided into all possible planes in all directions of the coordinate axes. Then the error can be written as:
−1 1 =1 =0 ( , ) = ∑
∑ 2( , , ) ; 2( , , ) = ( , , ), where – means axis (x, y or z) in which 2D texture was split, – number of planes for axis d in which the texture was split, , – -th plane in the d direction of the coordinate axis of the synthesized three-dimensional texture and 2 – is an error implying the difference between two-dimensional texture exemplars. For calculation of activation maps that are required by our error function, we use the same neural network as in VGG-19 paper.
textures is not an optimal approach since the Gram matrix itself poorly represents such aspects of the texture as the presence of structure, smooth inter-pixel transitions, and many others. For this reason, the existing exemplar-based methods [10, 11] of three-dimensional synthesis poorly work for input textures with complex structure: the resulting texture looks “broken”.
To solve these problems of the two-dimensional synthesis, in [2] it was proposed to generate a texture only not using the Gram matrix, but also a combination of several error components; moreover, an approach that allows preserving the structure of synthesized textures was presented in [3, 4, 8]. We have further developed idea from [2] to solve 3D synthesis problems and added three additional error components to train the generator proposed in [10]: using the first component, we have compared the structuredness of the generated texture and the sample. In fact, this error is the autocorrelation of feature maps of the given layers of the neural network, multiplied by the coefficient: , =
1 ( − | |)( − | |) ; ∈ [ 2
2 − , ] ; ∈ [ − where и – spatial sizes of feature maps ( ∗ = . Also for coefficient Gauss window could be used. The second introduced component of the error function increases the diversity of the generated textures and is the usual difference between feature maps: ,
1 ( , , ) = ∑
‖ ( , ) − ( )‖ ; ( , , ) = ∑‖ ( , ) − ( )‖ .
( , , ) = ∑‖ ( , ) − ( )‖ ; (9) (10) (11)
where – is a parameter of and algorithm.
Parameter values, used by us for texture synthesis
0.2, 0.2, 0.2, 0.2, 0.2 1 1 1
=0.5 =0.5*1e-6 =-1*1e-4 =-1*1e-3
=1e-3
1.
Textures were selected in the first group (Fig. 3,4) in order to find out how high-quality and original the textures synthesized by the generator, and also whether a proposed error components interferes with the synthesis. The textures were chosen, consisting of a monochrome background with some chaotic picture on top of it. These textures were chosen with the idea that the generator should work reasonably on this type of textures due to its simplicity, but the result may be too similar to the original texture or differ in random outbursts of colors atypical for this texture.
It can be seen from fig. 3-6 that the proposed method was able to show better results than it is base version [10]. Like [10], it works well with textures from the first group - the images obtained by the generator are new textures without any color outliers. Additional error components not only did preserve the stability of the synthesis, but also helped to broaden the already wide variety of generated textures, as well as improved the sharpness of pixel transitions (fig. 3, 4). If we look at the results of the second group, it can be seen that the proposed method preserves the structure better in textures containing structural patterns (fig. 5, 6). Thus, introducing additional components to the error helped to overcome the fundamental problem of poor preservation of structuredness inherent to the group of methods to which neural network synthesis belongs, and in the problem of synthesizing threedimensional textures.
[7] R. P éteri, S. Fazekas, M. J. Huiskes, DynTex: A Comprehensive Database of Dynamic Textures, 2010. URL: http://dyntex.univ-lr.fr/database.html [8] Y. Zhou, Z. Zhu, X. Bai, D. Lischinski, D. Cohen-Or, H. Huang, Non-Stationary Texture Synthesis by Adversarial Expansion, arXiv preprint (2018) arXiv:1805.04487 [9] A. Frühstück, I. Alhashim, P. Wonka, TileGAN: synthesis of large-scale non-homogeneous textures.
ACM Trans. Graph. 38(
[11] P. Henzler, N. J. Mitra, T. Ritschel, Learning a Neural 3D Texture Space From 2D Exemplars, in:
2020, pp. 8356–8364. [12] D. J. Rezende, S. Eslami, S. Mohamed, P. Battaglia, M. Jaderberg, N. Heess, Unsupervised Learning of 3D Structure from Images, Advances in neural information processing systems 29 (2016) 4996–5004. [13] J. Gwak, C. B. Choy, M. Chandraker, A. Garg, S. Savarese, Weakly supervised 3d reconstruction with adversarial constraint, in: International Conference on 3D Vision, 2017, pp. 263–272. doi:10.1109/3DV.2017.00038 [14] X. Yan, J. Yang, E. Yumer, Y. Guo, H. Lee, Perspective Transformer Nets: Learning Single-View
[15] J. S. De Bonet, Multiresolution sampling procedure for analysis and synthesis of texture images, in: Proceedings of the 24th annual conference on Computer graphics and interactive techniques.
doi:10.1145/258734.258882
[16] A. A. Efros, T. K. Leung, Texture Synthesis by Non-Parametric Sampling, in: Proceedings of the
seventh IEEE international conference on computer vision, 1999, pp. 1033–1038.
[17] A. A. Efros, W. T. Freeman, Image quilting for texture synthesis and transfer, in: Proceedings of
the 28th annual conference on Computer graphics and interactive techniques. ACM, New York, NY,
USA, 2001, pp. 341-346. doi: 10.1145/383259.383296
[18] L. Liang, C. Liu, Y.-Q. Xu, B. Guo, H.-Y. Shum, Real-time texture synthesis by patch-based
sampling, ACM Trans. Graph. 20(
[20] S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift, in: Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015, pp. 448–456. [21] X. Glorot, A. Bordes, Y. Bengio, Deep Sparse Rectifier Neural Networks, in: Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and