=Paper=
{{Paper
|id=Vol-2844/games2
|storemode=property
|title=Procedural 3D Terrain Generation using Generative Adversarial Networks
|pdfUrl=https://ceur-ws.org/Vol-2844/games2.pdf
|volume=Vol-2844
|authors=Emmanouil Panagiotou,Eleni Charou
|dblpUrl=https://dblp.org/rec/conf/setn/PanagiotouC20
}}
==Procedural 3D Terrain Generation using Generative Adversarial Networks==
https://ceur-ws.org/Vol-2844/games2.pdf
Procedural 3D Terrain Generation Using Generative Adversarial
Networks
Emmanouil Panagiotou∗ Eleni Charou
panagiotouemm@gmail.com exarou@iit.demokritos.gr
School of Electrical and Computer Engineering, National Institute of Informatics and Telecommunications, National
Technical University of Athens Centre for Scientific Research Demokritos
Greece Greece
Figure 1: Procedurally generated samples of satellite images.
ABSTRACT translation, to generate a plausible height map for every randomly
Procedural 3D Terrain generation has become a necessity in open generated image of the first model. Combining the generated DEM
world games, as it can provide unlimited content, through a func- and RGB image, we are able to construct 3D scenery consisting of
tionally infinite number of different areas, for players to explore. a plausible height distribution and colorization, in relation to the
In our approach, we use Generative Adversarial Networks (GAN) remotely sensed landscapes provided during training.
to yield realistic 3D environments based on the distribution of
remotely sensed images of landscapes, captured by satellites or CCS CONCEPTS
drones. Our task consists of synthesizing a random but plausible • Computing methodologies → 3D imaging; Neural networks;
RGB satellite image and generating a corresponding Height Map Adversarial learning.
in the form of a 3D point cloud that will serve as an appropriate
mesh of the landscape. For the first step, we utilize a GAN trained KEYWORDS
with satellite images that manages to learn the distribution of the deep learning, general adversarial networks, satellite imagery, pro-
dataset, creating novel satellite images. For the second part, we need cedural generation, gaming, 3D point cloud, digital elevation models
a one-to-one mapping from RGB images to Digital Elevation Models
(DEM). We deploy a Conditional Generative Adversarial network
(CGAN), which is the state-of-the-art approach to image-to-image
∗ Corresponding author
GAITECUS0, September 02–04, 2020, Athens, Greece
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).
1 INTRODUCTION that error to previous layers, updating the trainable weights accord-
Procedural content creation has been used in the past by game ingly. Compared to standard feedforward neural networks, CNNs
developers, as it can offer increased gameplay variety and replaya- are able to make strong hypothesis regarding the nature of the
bility for the player, as well as lower budgets for gaming companies. images as they take the 2D structure into account, thus using much
Renowned games of different genres such as the Borderlands [19] fewer connections and parameters, leading to faster training times
and Civilization [20] series, Minecraft [11] and No Man’s Sky [21], [9].
apply analogous techniques. Procedural generation is an emerging
research field on Artificial Intelligence (AI) & gaming, leading to 2.2 Generative Adversarial Networks
various new state-of-the-art approaches [3, 23]. In most cases, de- Generative Adversarial Networks (GAN) [1] constitute a general
velopers create procedural characters, dungeons or landscapes by framework for training generative models, i.e. models that can pro-
using predefined templates that can randomize some aspects of the duce samples, not only differentiate between them. GANs consist
generated object. As discussed in [10], games should take advan- of a generator G and a discriminator D, both modeled as artificial
tage of real-world information available on the internet. For our neural networks. The generator is optimized to reproduce the true
approach we generate random images that follow the distribution data distribution 𝑝𝑑𝑎𝑡𝑎 , which can be fixed to the distribution of
of real remotely sensed imagery. Particularly, we require a function interest, by generating images (or any form of data) that are difficult
for the discriminator to differentiate from the real images, namely
𝑃 :𝑍 →𝑋
the actual data distribution 𝑝𝑑𝑎𝑡𝑎 . Simultaneously, the discrimina-
where 𝑍 is random noise and 𝑋 is the generated image. To add a tor is tasked with differentiating real images from synthetic data
dimension of height to each pixel of the generated image, a one- generated by G. Their training procedure is a minimax two-player
to-one mapping 𝐺, generates a 3D point cloud or Digital Elevation game with the following objective function:
Model (DEM) for each input tile 𝑋 . Specifically,
𝐺 :𝑋 →𝑌 min max 𝑉 (𝐷, 𝐺) =𝑥∼𝑝𝑑𝑎𝑡𝑎 (𝑥) [log 𝐷 (𝑥)]+𝑧∼𝑝𝑧 (𝑥) [log (1 − 𝐷 (𝐺 (𝑧)))]
𝐺 𝐷
where 𝑋 is the domain of images produced by 𝑃 and 𝑌 that (1)
of DEMs. Both tasks require a rule-based approach, as the gener- where 𝑧 is a noise vector sampled from a prior noise distribution
ated input images, as well as the resulting one-to-one mappings of choice 𝑝𝑧 , usually a uniform or a normal distribution, and 𝑥 is a
are infinite. Obviously, both systems are impossible to "hard-code", real image, from the data distribution 𝑝𝑑𝑎𝑡𝑎 . [1] prove that, given
therefore AI or Machine Learning (ML) models have to be imple- enough capacity, the generator can learn to replicate the true data
mented, as they can learn such rules in an automated, data-driven distribution.
manner. In particular, Deep Learning (DL), the data-intensive ver-
sion of ML, has recently been proven to be useful for many diffi- 2.3 Conditional Generative Adversarial
cult problems. Especially in image processing tasks for computer Networks
vision [5, 9, 22], specific DL algorithms are the go-to solutions.
As suggested in [1] and first examined in [12], CGANs can extend
Consequently, we propose a DL method for procedural 3D scenery
GANs by incorporating additional information, like a class label
generation that is data driven and relies solely on real remotely
or, analogous to our case, extracted features, in effect conditioning
sensed imagery, with no need of any input from the developer. The
the generator and the discriminator to it. Denoting the additional
model succeeds in replicating the input data distribution, generat-
conditioning variable as 𝑐, we can substitute 𝐷 (𝑥) and 𝐺 (𝑧) from
ing images and 3D representations of increased variation and high
Equation 1 with 𝐷 (𝑥 |𝑐) and 𝐺 (𝑧|𝑐), whereas the rest of the formu-
quality. The code for our work has been made publicly available at
lation remains the same:
https://github.com/Panagiotou/Procedural3DTerrain.
2 DEEP LEARNING TECHNIQUES FOR IMAGE min max 𝑉 (𝐷, 𝐺) =𝑥∼𝑝𝑑𝑎𝑡𝑎 (𝑥) [log 𝐷 (𝑥 |𝑐)]+𝑧∼𝑝𝑧 (𝑥) [log (1 − 𝐷 (𝐺 (𝑧|𝑐)|𝑐))]
𝐺 𝐷
PROCESSING (2)
In this section, we provide necessary context to the information By conditioning on 𝑐, we can control the quintessence of the
discussed throughout our research paper. output of the generator, allowing the noise 𝑧 to add background
information, pose, etc [4, 17, 24, 25].
2.1 Typical Convolutional Architecture
A Convolutional Neural Network (CNN) is a (deep) neural network 3 DATASET
consisting of an input layer, multiple hidden layers and an output In order for a CNN architecture to be trained, a large-scale dataset
layer. The first layer, expects an image as input, which is passed to is imperative, as well as computing power to process it, preferably
the next layers. Every hidden layer is comprised of convolutional with the parallel processing capabilities of a Graphics Processing
layers that convolve the input by applying a dot product with a ker- Unit (GPU). This especially holds in our case, where the objective is
nel consisting of trainable weights. The resulting output is passed to train a GAN architecture for generating a vast variety of random
by a pooling layer that reduces the input dimensions for the next images. Our second task consists of performing an image-to-image
layer. The output layer computes the error of the predicted output translation from those generated images to their corresponding
in relation to the expected ground truth values and backpropagates DEMs. During this process the DEM is interpreted as a single band
(grayscale) image. Evidently, a dataset of pairs of RGB satellite The overall process is graphically presented in Figure 3. Some
images and their corresponding DEM images is needed. As we pairs of the dataset can be seen at Figure 2. As a preprocessing step,
were unable to acquire data containing both RGB and DEM im- we project the DEMs to the [−1, 1] range, as each tile was scaled
ages, we decide to build our own. To be more precise, a large area to the [−1, 1] range according to the global minimum-maximum
over Greece was selected as our region of interest (ROI). The DEM of the entire dataset.
images corresponding to our ROI are provided by ALOS Global
Digital Surface Model "ALOS World 3D - 30m (AW3D30)" [7] and
can be granted with a request to the respective owners. We then
split the DEMs into smaller tiles and, for each tile, a script obtains
the corresponding RGB tile. In particular, the program extracts a
GeoJSON polygon from the georeferenced DEM tile and feeds it
to the Google Earth Engine API [2], which is publicly available.
This, then, returns the true color bands [TCI_R, TCI_G, TCI_B]
Sentinel-2 MSI, which, when stacked, yield the requested RGB satel-
lite image corresponding to the input DEM. To get the final dataset
we reshape our data so that all tiles are 256 × 256 pixels.
Figure 3: Flow chart of the satellite-imagery dataset collec-
tion process using the Google Earth Engine API. The outer-
most points of the georeferenced DEM are selected as the
boundary for the GeoJSON Polygon, which when fed into
the API returns the corresponding RGB satellite image.
(a) Satellite Images
4 GENERATIVE ADVERSARIAL NETWORK
FOR SATELLITE IMAGE GENERATION
As aforementioned, the first step in procedurally producing random
3D landscapes, is generating random images that mimic the real
satellite images of the dataset. While attempts for lower resolution
image generation [1, 15] have been successful, researchers discov-
ered a difficulty in convergence mainly for higher resolutions. This
effect called "mode collapse" occurs when the discriminator, at some
point, wins the minimax game resulting in non-convergence of the
generator, who starts producing similar results for every input sam-
ple. In our case, we choose to construct images of size 256 × 256.
Therefore, we choose to implement a technique of progressive grow-
ing GANs (ProGAN) introduced in [8]. This architecture, allows
training to occur in multiple stages. Instead of training all layers
of the generator and discriminator models, ProGANs are trained
one layer at a time, leading to exponential growth of the generated
images on every step. This method, proves to be very effective in
(b) DEMs stabilizing the training process and reducing its duration, leading
the generator to convergence and producing images of high reso-
Figure 2: Example (a) satellite images and (b) their corre- lution at the same time. The increase in resolution is achieved by
sponding DEM tiles over different locations of Greece. adding new layers to both networks as seen in Figure 4.
Figure 5: The generator architecture of choice: U-net [18].
It consists of an encoder that downsamples the input im-
age using convolutional blocks up until the bottleneck layer.
Thereafter, deconvolutional blocks upsample the image to
the desired dimensions. The skip connections, denoted by
pointed arrows between corresponding layers of the en-
coder and the decoder, facilitate training by providing cru-
Figure 4: Flow chart of the training process. Both the Gen- cial lower-level information from the encoder to the de-
erator and the Discriminator start with a low resolution of coder. Given that the input and the output have the same
4 × 4. Size is advanced exponentially, until the target distri- low-level structure, these low-level features serve as the can-
bution of 256 × 256 is reached. vas that guides the decoder in the generation of the final out-
put.
The discriminator model is a binary classifier, deciding whether
a given image (e.g. DEM) has been produced by the generator, or
belongs to the real images provided by the dataset. Deep CNNs
have been heavily tested and therefore proven to work on image
classification tasks [16]. In our case, a PatchGAN [6] is used. The
The weights of all previous layers, remain trainable during this main difference is that the traditional CNN architecture would
process and for the model to avoid shocks during this transition, come to a decision based on the whole input image, whereas the
new layers are faded in gradually. This process of fading in a new PatchGAN maps the 256 × 256 image, in our case, to a square array
layer, is controlled by a parameter 𝛼 ranging from 0 to 1 over the of outputs. Each output "pixel" signifies whether the corresponding
course of multiple iterations, producing a weighted sum of the two patch is real or fake.
last layers of the generator. The discriminator can be regarded as a
symmetrical copy of the generator. Input images are either "fake"
images synthesized by the generator or real images of the dataset,
obviously scaled down to the current training resolution. Through
a series of convolutional layers, the image is downscaled, until the
last layer, where a boolean decision is returned.
5 CONDITIONAL GENERATIVE
ADVERSARIAL NETWORK FOR
ELEVATION PREDICTION
Following [6], we use the pix2pix architecture to train the CGAN
framework. In particular, we use an encoder-decoder architecture,
described as U-net in [18] for the generator. This model, first down-
samples the conditioning input (e.g. satellite) image down to a
bottleneck layer using a series of convolutional layers. Afterwards,
through a series of deconvolutions, roughly the inverse operator of
the convolution, the images are upsampled, decoding the bottleneck
code to the size of the output image. Every convolutional layer is
connected with a skip connection to its respective deconvolutional Figure 6: The discriminator architecture of choice: Patch-
layer, helping the model to converge during training since it skips GAN [6]. The discriminator decides whether its input is
some layers by feeding the output of one layer as the input to next from the true data distribution based on local information
layers [26], which facilitates training, provided the global, low-level by concentrating on the fidelity of individual image patches.
structure is the same between input and output, as is the case in Convolutional and pooling layers are applied to reduce the
our task. The architecture of the U-net can be seen in Figure 5. dimensions of the input images.
The final decision for the whole image is derived by averaging
over all the individual patches. Using a Patch-based approach for
the Discriminator, compared to a traditional CNN architecture for
binary image classification, has proven to encourage high frequency
crispiness in the resulting images [6]. The PatchGAN architecture
can be seen in Figure 6.
The task of predicting plausible DEMs for input remotely sensed
imagery , as well as, model evaluation and accuracy have been
addressed thoroughly in our previous work [13].
6 RESULTS
(a) Satellite Images
We first present our results of the ProGAN model Figure 8a. It is
evident that random RGB satellite images of great resolution and
variety are being generated. The DEMs produced by the CGAN
model, presented in Figure 8b, render a plausible representation in
relation to the input images, as well as the data distribution of DEMs
provided during training. We observe that the ProGAN model, by
progressively growing the size of the output image, has learned
to generate sharp results that imitate images that are present in
our dataset. Various basic elements like river banks, islands with
greener water near the surface and snow, are present. Likewise, the
CGAN model produces detailed and accurate DEMs, resulting in
plausible 3D representations, Figure 7.
(b) DEMs
Figure 8: Procedurally generated (a) satellite images and (b)
respective DEM tiles produced by the CGAN . Images pro-
duced are diverse and at the target resolution of 256 × 256.
7 DISCUSSION-FUTURE WORK
While individual results of our approach presented in Figures 8 and
7, are remarkable, an emerging problem is choosing neighboring
tiles. In particular, while game content is generated and if the game
is infinite-world, every tile needs to have 8 neighboring tiles. This
process of choosing appropriate tiles, is left for future research,
but one approach could be using images produced by latent codes
close to the one which produced the center tile. Close latent points,
in our case, are similar noise vectors, which therefore produce
similar images. One can then create a linear interpolation between
a starting and a target image, like the one presented in Figure 9.
Figure 7: A 3D visualization of the generated landscapes pro-
duced by both models. Figure 9: A sparse interpolation in Latent Space.
A more lightweight solution for producing 3D landscapes intro- [15] Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representa-
duced in [13] is to use a single CGAN model, solving the inverse tion learning with deep convolutional generative adversarial networks. arXiv
preprint arXiv:1511.06434 (2015).
problem i.e., train the inverse operator, 𝐺 −1 , to predict the surface [16] Waseem Rawat and Zenghui Wang. 2017. Deep convolutional neural networks
coloration, meaning the RGB image, conditioned on a DEM. In this for image classification: A comprehensive review. Neural computation 29, 9 (2017),
2352–2449.
case, a random 256 × 256 DEM tile is sampled from a Perlin noise [17] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele,
distribution [14], which is especially suited for generating plausible and Honglak Lee. 2016. Generative adversarial text to image synthesis. arXiv
landscapes with peaks and valleys. preprint arXiv:1605.05396 (2016).
[18] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional
networks for biomedical image segmentation. In International Conference on
Medical image computing and computer-assisted intervention. Springer, 234–241.
[19] Borderlands (series). 2009–2019. PlayStation 3. (2009–2019).
[20] Civilization (series). 2009. Microsoft Windows. (2009).
[21] No Man’s sky. 2018. PlayStation 4. (2018).
[22] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir
Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.
Going deeper with convolutions. In Proceedings of the IEEE conference on computer
vision and pattern recognition. 1–9.
[23] Julian Togelius, Georgios N Yannakakis, Kenneth O Stanley, and Cameron Browne.
2011. Search-based procedural content generation: A taxonomy and survey. IEEE
Transactions on Computational Intelligence and AI in Games 3, 3 (2011), 172–186.
[24] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei
Figure 10: The well known technique of generating DEMs Huang, and Dimitris N Metaxas. 2017. Stackgan: Text to photo-realistic image
synthesis with stacked generative adversarial networks. In Proceedings of the
with random Perlin noise, is enhanced by adding plausible IEEE international conference on computer vision. 5907–5915.
colors to the random DEM, using the trained inverse CGAN [25] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei
model. Huang, and Dimitris N Metaxas. 2018. Stackgan++: Realistic image synthesis with
stacked generative adversarial networks. IEEE transactions on pattern analysis
and machine intelligence 41, 8 (2018), 1947–1962.
[26] Xiaobin Zhu, Zhuangzi Li, Xiaoyu Zhang, Haisheng Li, Ziyu Xue, and Lei Wang.
In conclusion, an idea left for future work, is to implement a 2018. Generative Adversarial Image Super-Resolution Through Deep Dense
global model combining the scopes of both models, e.g. generating Skip Connections. In Computer Graphics Forum, Vol. 37. Wiley Online Library,
289–300.
random satellite imagery while producing a plausible DEM repre-
sentation. This model will have to minimize a combined loss for
both problems, probably leading to difficulty in convergence, but
would likely yield more realistic and robust results.
REFERENCES
[1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial
nets. In Advances in neural information processing systems. 2672–2680.
[2] Noel Gorelick, Matt Hancher, Mike Dixon, Simon Ilyushchenko, David Thau, and
Rebecca Moore. 2017. Google Earth Engine: Planetary-scale geospatial analysis
for everyone. Remote sensing of Environment 202 (2017), 18–27.
[3] Daniele Gravina, Ahmed Khalifa, Antonios Liapis, Julian Togelius, and Georgios N
Yannakakis. 2019. Procedural content generation through quality diversity. In
2019 IEEE Conference on Games (CoG). IEEE, 1–8.
[4] Irina Higgins, Loic Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles
Blundell, Shakir Mohamed, and Alexander Lerchner. 2016. Early visual concept
learning with unsupervised deep learning. arXiv preprint arXiv:1606.05579 (2016).
[5] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J
Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x
fewer parameters and< 0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016).
[6] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-
image translation with conditional adversarial networks. In Proceedings of the
IEEE conference on computer vision and pattern recognition. 1125–1134.
[7] JAXA-EORC. 2016. ALOS Global Digital Surface Model "ALOS World 3D - 30m"
(AW3D30). http://www.eorc.jaxa.jp /ALOS/en/aw3d30/index.htm (2016).
[8] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Pro-
gressive Growing of GANs for Improved Quality, Stability, and Variation.
arXiv:1710.10196 [cs.NE]
[9] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifica-
tion with deep convolutional neural networks. In Advances in neural information
processing systems. 1097–1105.
[10] Antonios Liapis. 2018. Real-world data as a seed, The Procjam Zine,. 3 (2018),
35–40.
[11] Minecraft. 2009. Xbox 360. (2009).
[12] Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets.
arXiv preprint arXiv:1411.1784 (2014).
[13] Emmanouil Panagiotou, Georgios Chochlakis, Lazaros Grammatikopoulos, and
Eleni Charou. 2020. Generating Elevation Surface from a Single RGB Remotely
Sensed Image Using Deep Learning. Remote Sensing 12, 12 (2020), 2002.
[14] Ken Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics 19, 3
(1985), 287–296.