Introduction
The prediction of future frames has several applications in autonomous
decision-making areas that include; self-driving cars, social robots and video completion [
10
]. For
example, a SocialGAN [
4
] determines plausible and socially acceptable walking
trajectories of people, thus, aiding in the navigation of human-centric environments. GANs
([
1
], [
4
], [
5
], [
6
], [
9
]) have been a popular approach to training spatio-temporal models
for future frame prediction. The constituent components of a GAN is a generator and a
discriminator, engaged in a minimax game [
3
]. GANs, however, are difficult to train;
and are susceptible to mode collapse. In transformation space, the generator extracts
transformations between adjacent input frames. It subsequently predicts a future
transformation and applies it to the last frame of the input to generate the next frame and so
forth. The source of variability is modelled and, thus, the need to store low level details
of the input is eliminated. The resultant model requires fewer parameters which
simplifies learning. Furthermore, the spatial data of the input is conserved.
To model spatio-temporal relationships in video data, networks include either CNNs,
RNNs or both. The standard for sequential modelling tasks is RNNs, such as LSTMs,
due to its ability to represent long term temporal dependencies. A CNN that exhibits a
similar efficacy is the Temporal Convolutional Network (TCN). A TCN in conjunction
with a dilated CNN to model temporal and spatial dependencies respectively was
implemented by [
9
]. A similar approach was undertaken by [
1
], with a PGGAN modelling
spatial dependencies instead. Another attempt at sequential modelling utilizing CNNs
[
8
] was an architecture in which a network was replicated through time. The resultant
model was a ‘peculiar RNN’ as parameters were now shared across time whilst still
convolving spatial data. A CNN-LSTM architecture was implemented by [
6
] to predict
future frames of synthetic video data. These aspects were later united by [
7
] into a single
network, a convolutional LSTM (ConvLSTM). A stacked ConvLSTM, coupled with a
Spatial Transformer Network (STN) [
2
], addressed the problem of future frame
prediction and determined the state of motion of a robot arm. The representation of motion is
improved by models that operate in transformation space ([
2
], [
8
], [
9
]). Such a model,
a CGAN [
9
] was evaluated using a Two-Alternative Forced Choice (2AFC) test. The
generated video was preferred only 30.6% of the time over its ground-truth counterpart.
3
Proposed Model
In a bid to address the issues of motion representation, photorealism and plausibility of
generated frames, this research proposes the implementation of a CGAN. The
discriminator of the CGAN receives the context frames coupled with alternatively ground truth
future frames or generated future frames and is only deceived by sequences of frames
that exhibit plausibility. A mini-batch standard deviation layer is added to one of the
last layers of the Progressively Growing Network (PGN) discriminator; aiding in the
prevention of mode collapse. The generator comprises of 7 stacked ConvLSTMs,
similar to [
2
], and preserves spatial data whilst modelling the complex dynamics of the
data. Hidden Layer5 parameterizes a modified STN and the output of ConvLSTM5 is
a predicted affine transformation matrix for each separate ‘good feature’ in the frame.
The STN is modified to determined points by the Shi-Tomasi Corner Detection
algorithm for which transformations are then predicted. The model also predicts a
compositing mask over each transformation. The generated frame is reconstructed by applying
predicted affine transformations, merged by masking, to the last input frame.