A Generative Model is a powerful way of learning any kind of data distribution using unsupervised learning and it has achieved tremendous success in just few years. Though there are several approaches to design information systems for generating synthetic data, wich are referred to as Deep Generative Model (DGM). Since then, DGM has become a trending topic both in academic literature and industrial applications. It is also receiving increasing attention in machine learning competitions. This paper aims to provide an overview of the current progress towards DGM, as well as discussing its various applications and open problems for future research. Moreover, we discuss some research we conducted during last years that may extend the existing state of the art approaches in synthetic data generation or improving existing deep models.
Copyright c by the paper's authors. Copying permitted for private and academic purposes.
All types of generative models aim at learning the true data distribution of the training set so as to generate new data points with some variations. But it is not always possible to learn the exact distribution of our data either implicitly or explicitly and so we try to model a distribution which is as similar as possible to the true data distribution. For this, we can leverage the power of neural networks to learn a function which can approximate the model distribution to the true distribution.
Two of the most commonly used and e cient approaches are Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN). VAE aims at maximizing the lower bound of the data log-likelihood and GAN aims at achieving an equilibrium between Generator and Discriminator. Many researches attempt to compile a uni ed view: new formulation of GANs and VAEs, and linked back to the classic variational inference algorithm and the wake-sleep algorothm. 2.1
The variational autoencoder [Kingma13, Rezend14] is a directed model that uses learned approximate inference and can be trained purely with gradient-based methods. An autoencoder can be used to encode an input image to a much smaller dimensional representation which can store latent information about the input data distribution. The variational autoencoder approach is theoretically pleasing and simple to implement. It also obtains excellent results and is among the state-of-the-art approaches to generative modeling.
One of the main features is that samples from VAEs trained on images tend to be somewhat blurry. The causes of this phenomenon are not yet known. The key idea of VAE are shown in Fig. 1.
z ~ N(0, I)
q
X
The primary objective is to model the data X with some parameters which maximizes the likelihood of training data X. In short, we are assuming that a low-dimensional latent vector has generated our data x(x 2 X) and we can map this latent vector to data x using a deterministic function f (z; ) parametrized by theta which we need to evaluate. During generative process, our aim is to maximize the probability of each data in X. 2.2
Generative adversarial networks, or GANs [Goodfellow16], are another generative modeling approach based on di erentiable generator networks. Adversarial training has completely changed the way we train the arti cial neural networks. GAN dont link with any explicit density estimation like VAE. GAN is based on game theory approach with an objective to nd equilibrium between the two networks: generator and discriminator. The aim is to sample from a simple distribution and then learn to transform this noise to data distribution using approximators such as neural networks. This approach is shown in Fig 2
noise z
We can formulate learning in GAN as a zero-sum game, in which a function v( (g); (d)) determines the payo of the discriminator. During learning, each player attempts to maximize its own payo, so that at convergence g = arg ming maxd v(g; d). One of the earliest model on GAN employing Convolutional Neural Network (CNN) is Deep Convolutional Generative Adversarial Networks (DCGAN) [Radf17]. 3
Unsupervised learning is a next frontier in arti cial intelligence. One of the main advantages of Generative Models is a possibility of the training in semi-supervised manner. Such models can be applied for solving complex problems: text-to-image translation, synthetic images generation, solving problems with multimodal data distribution, drug discovery, visual marks retrieval from images. DGM is a way to improve existing discriminative models. GANs help to solve the one of the main challenges in deep learning: huge amount of labelled data. These models help in building a better future for machine learning. [Smol86] P. Smolensky Information processing in dynamical systems: Foundations of harmony theory. Parallel distributed processing: Explorations in the microstructure of cognition, MIT Press, Cambridge, MA, (1986).
A fast learning algorithm for deep belief nets. Neural [Salakh09] G. E. Hinton, R. Salakhutdinov Deep Boltzmann Machines. To appear in Arti cial Intelligence and
Statistics. 2009. [Goodfellow16] Goodfellow Nips 2016 tutorial: Generative adversarial networks. NIPS 2016, arXiv:1701.00160, 2016. [Zarem14] W. Zaremba, I. Sutskever and O. Vinyals Recurrent Neural Network Regularization. arXiv:1409.2329 [cs.NE]. 2014.