We propose a framework that combines Generative Adversarial Networks and Evolutionary computation to perform Data Augmentation on small datasets in order to improve the performance of image classifiers trained via supervised learning. In this work, we attest the viability and potential of this framework for real-world problems. The framework is composed of a generator module that uses Generative Adversarial Networks to generate samples from a dataset. It employs an Evolutionary Computation approach to evolve sets of images from the latent space. The fitness function is based on the dissimilarity of the subsets generated by the Generative Adversarial Network. A Supervisor module handles the generated samples and chooses which set should be added to the training dataset. To test the framework, we explore the Human Sperm Head Morphology dataset, a bio-medicine multi-class problem with a small number of samples that provide a challenge to the different supervised classification approaches. We deploy the framework to create an augmented dataset to train a classifier, and after the training, we compute the performance on the test set. We compare with classifiers trained using the base datasets without having the generated samples. Overall, with the preliminary tests, we can improve the performance of the classifiers by up to 4% and on average by 1%, showing the viability and potential of our approach.
With the evolution of technology and computer capabilities, Machine Learning has seen significant improvements in recent years. Like so, it became much easier to build and apply larger neural networks, such as Deep Neural Networks, to solve real-world problems. It also became possible to build Deep Generative Models which produce synthetic data by learning from already existing data.
While the availability of data has also been accompanying the evolution of technology, there are still many problems that lack enough data to allow Machine Learning algorithms to be viable solutions for them. The performance of Machine Learning algorithms depends not only on the capability of the model used but also on the quality of the dataset used in the training of the model. This means that training a model with a bad dataset will, most likely, lead to poor performance results. As such, improving the quality of the datasets through Data Augmentation may be a way to improve the performance of the algorithm. With this in mind, is it possible to use generative models to enhance the quality of existing datasets, consequently improving the quality of machine learning algorithms?
One of the ways to train generative models is by using
Generative Adversarial Networks. These frameworks are more often used to
produce very realistic images that follow the distribution of the
training dataset [
As a further matter, it is also essential to address the generation of
new samples. Even though a capable model is important in the
generation of better images, there is also another variable that impacts
the instances generated, which is the latent space. It is unique to each
generative model and hides underlying patterns in itself. Usually, to
generate an image, a vector from the generative model’s latent space
is chosen at random for input, which means that there is no
knowledge about the output. Since the images generated depend on the
input given to the generator, the exploration of the latent space may
reveal ways to control the output through the selection of input
vectors by certain criteria. This way, we can also assure the quality of
the images that will be added to the dataset and their relevance to
help with the problem at hand. For instance, we might want to ensure
that we are not adding redundant samples to the training set. Like so,
performing random Data Augmentation should have a higher chance
of undermining the performance of the algorithm, which means that
ideally, we should prefer an approach of a supervised generation of
instances. Bearing this in mind, we chose to perform this supervision
by exploring the latent space using Evolutionary Computation. Using
a Genetic Algorithm, we evolve sets of latent vectors which optimize
a specific criterion such as the diversity of images in the set. This
framework for latent space exploration was explored in [
The remaining of the paper goes as follows: in the next Section, we explain our approach to the problem and the framework used to solve it (Section 3). In Section 4, we describe the experimental setup and analyse and discuss the results. In Section 5 we draw overall conclusions.
Generative Adversarial Networks are generative models that are
trained through a face-off between a generator and a discriminator,
mostly used to train a generator that can produce realistic images.
The generator is given a noise vector to produce new images,
usually, a high dimensional vector that is randomly sampled from a
distribution, for example, a gaussian distribution, called the prior. The
high dimensional space is called latent space. Some work has already
been made to explore the latent space of generative models, and not
only with Generative Adversarial Networks, which is the framework
that we will use in this work. For instance, latent space exploration
was performed in Kernel Principal Component Analysis [
Evolutionary Computation has also been used in some works in
order to evolve images. For instance, evolving master print templates
[
In this paper, we propose a framework that combines Generative Adversarial Networks and Evolutionary computation to perform Data Augmentation on small datasets in order to improve the performance of image classifiers trained via supervised learning.
For the framework, there are 3 fundamental pieces: (i) a
classifier responsible for the classification task, discriminating images into
classes; (ii) a generator, responsible for generating new images, from
an array of the latent space; (iii) a supervisor, responsible for
managing the generation images through the exploration of latent space (as
in [
The performance of classifier will measure the performance of this framework. Therefore, we will be looking into comparing the performance of the classifier after the baseline training - using only the original dataset - against the performance of the classifier after the supervised augmented training - with selective addition of synthetic images to the training set. This way, we will be able to assess the quality of our approach and guide the progress of research. 3.2
The generator will be obtained through a Deep Convolutional
Generative Adversarial Network. These make use of Deep Convolutional
layers that better explore space correlation in images [
Lastly, the Supervisor, which is crucial to the optimization of the
training dataset. By adding random images with no criteria, there is
no way to ensure if these are relevant to the solution of the problem.
The introduction of flawed and redundant images might end up
undermining the performance of the classification algorithm [
In order to evaluate our approach, we performed several tests. The conditions on which these tests were carried out are presented in this section. 4.1
I order to test our hypothesis we will be using the Human Sperm
Head Morphology dataset (HuSHeM) [
Each image has a original dimensions 131x131x3, but in the experiments we will be working with in dimensions 132x132x1. 4.2
The classifier module allows the assertion of the experiment results. The model used was an off-the-shelf model that only required training since the optimization of the model was one of our objectives for this work. The parameters used for the training of the classifier are present in Table 1. The number of epochs chosen ensures that the classifier reaches a point of plateau for the original dataset, where there is no gain in performance. This helps to verify the quality of our solution. On another note, the training also included cross-validation for every test, which means both tests with the original dataset and augmented datasets. The generators allow for the creation of new samples to perform data augmentation. A generator is obtained through unsupervised training of a Deep Convolutional Generative Adversarial Neural Network. The model used for the discriminator, as the classifier, is an off-theshelf model, to which we only added 2 extra layers of convolution. As for the discriminator, the model used was the same as the classifier. The training is performed in each individual class, which means that in this case, in particular, we are going to need 4 different generators to produce samples for each class [Figure 2]. Each generator was trained using the 40 class corresponding samples in the training set. The training parameters were set, as shown in Table 2. The core step of this work is the supervision of the generation of samples. This is what is going to allow the optimization of the process of performing Data Augmentation and ensure the best results possible. For this, we decided to use Evolutionary Computation, namely a Genetic Algorithm, to explore the latent space of the generators with the intent of finding sets of algorithms that optimize a certain criterion. In this case, specifically, we are looking into maximizing the diversity of the dataset. The supervision is performed for a single generator, or single class, which means that in this problem, we are going to use 4 supervisors to evolve 4 different sets of images that will be added to the original set. The parameters used in the genetic algorithm were as defined in Table 3.
Parameter Population size Number of generations Genotype length Elite size Selection method Tournament size Crossover operator Crossover rate Mutation operator Mutation distribution Mutation rate per gene
T is the set resulting of concatenation of the images from the individual (I) with the images from the original dataset(O). C is the centroid of the set T . F is the fitness of the individual calculated through averaging the similarities measured using the Normalized Cross-Correlation(NCC). The Calculation of the similarity metric is as follows:
N CC(A; B) =
P((A pP(A
B) A) (A P(B
B))
B) The corresponds to the Hadamard product between two images.
On a last note, since the fitness function measures similarity instead of diversity, the objective of the algorithm is set to minimization. In the end, we should end up with a set of images that are more diverse, than if we just picked a randomly. 4.5
In order to test our framework, we performed a comparison between the performances of the classifiers before and after performing Data Augmentation. The evaluation of each classifier in the crossvalidation was performed in the test dataset, where several metrics were measured, namely Accuracy, Precision, Recall, F1 score, Area Under Receiving Operator Characteristic Curve (AUROC) and Average Precision. Each test was performed 5 times with different seeds. In the following results we will be presenting the mean across these 5 repetitions. Note that the initialization of the weights is the same between Model-X and Seed-X (e.g. Model-0 and Seed-0), Seed-0 differs on the seed used to generate the augmented dataset and, of course, the existance of augmented instances.
The first tests were performed with the original dataset. By looking at Figure 3 we can observe that at 250 epoch the training has already reached a plateau in terms of accuracy, which means that it will most probably not get any benefits from further training since it will tend to overfit. We were also able to find the performance that our solution should be able to overcome [Table 4.5].
The next step was building the sets of images of images to be added to the original dataset and train with the augmented dataset. For this experiment it was decided to test a dataset composed of 50% original images and 50% synthetic images. As such, for each class we generated and evolved sets of 40 images. The selection of these sets was repeated for every repetition of the test with the classifier which was performed 5 times with different seed, similarly to what was done with the baseline test.
First, by analysing the line of evolution in Figure 4 we can see that the values of fitness of the best individuals do not have a great variation between the first and the last generations. All values, for every class, sit on an interval of 0.01 between, 0.99 and 1. This means that the similarity between the images in this dataset, for this metric, is really high and that it promote a good evolution. However, if we take look at Figure 5 which puts side-by-side the best individual of the first generation (top) against the best individual of the last generation (bottom) in the evolution of a set of the class ”Amorphous”, we can (1) (2) (3) (4) argue that the latter has in fact, from a subjective perspective, more visual diversity than the first.
The last step is the training of the classifiers with the augmented datasets. By analysing the training curve in Figure 6 we can see that at epoch 250 the training also reaches a plateau accuracy-wise, meaning that further training would not help better the performance.
Looking at the test results [Table 4.5], we can see that the performance of the classifiers trained with the augmented dataset was, on average, better for every metric. Although it would be necessary to perform more tests to verify the benefit of our solution, this shows that our approach might indeed be a way to improve datasets and consequently the performance of Classifiers. Each Seed-X represents a classifier that was trained by a subset from the Evolutionary Computation process using a different random generator seed. We can observe that in 4/5 seeds we are able to improve beyond the average performance of the original baseline classifier. We have one Seed that improves up to 4% over the baseline average. One of the seeds hindered the performance of the classifier but on average we have improvements over all the metrics when compared with the trained models with different initialization weights. 5
We explored an approach that uses Generative Adversarial Networks for Data Augmentation to improve the performance of a supervised classifier applied in a real-world problem. The underlying idea is to explore the latent space of the generative model using Evolutionary Computation to generate sets of instances to be used in the training dataset of a supervised classifier. Since arbitrarily adding instances to the dataset could hinder the performance of the classifier that is denoted as Model-X and each model trained with the augmented dataset from the Evolutionary process is denoted as Seed-X. Augmented is the average of the 5 seeds and Original is the average of 5 models trained. 0.593 0.575 0.571 0.571 0.579 0.589 0.589 0.539 0.596 0.618
Prec 0.623 0.610 0.648 0.599 0.621 0.641 0.633 0.645 0.632 0.586 0.642 0.663 0.598 0.580 0.569 0.575 0.580 0.587 0.596 0.534 0.604 0.629
Metrics
F1
Avg-Prec 0.719 being trained, we rely on a Supervisor module that selects the best set based on different criteria.
We instantiate this framework in a real-world application problem the Human Sperm Head Morphology dataset has a proof of concept. Due to the small number of instances we can categorize it as small data dataset, which presents an opportunity to explore Data Augmentation approaches. We created a baseline classifier with the provided data for comparison with the classifiers created by our framework. We used a Genetic Algorithm to evolve sets of latent space vectors that generated sets of images. We used the normalized crosscorrelation similarity metric to calculate the dissimilarity among the sets and used the average value to assign fitness to each one. Overall we were able to guide evolution and generated dissimilarity subsets. The best subset from the last population of the evolutionary algorithm was used to augment the training dataset of the classifier. The classifier was then trained with the synthetic and with a base subset of instances. We used cross-validation to compute performance metrics. Overall the results show that we can increase the performance of the classifier. For example, we were able to raise accuracy by 0.8% and the f1-score by 1.2%. Although more tests are needed to verify this conclusion and even to improve the quality of the solution, it is a first step and a proof of concept of the potential of such an approach.
Future work may include testing with different proportions between original images and generated images in augmented datasets, and even testing on training sets composed of generated images only. Also, we may even test different datasets, use different similarity metrics or even improve the supervision algorithm with the inclusion of other techniques. Finally, we may also look into comparing this approach to other data augmentation approaches.