Advances in deep learning have led to promising progress in inferring graphics programs by de-rendering computergenerated images. However, current methods do not explore which decoding methods lead to better inductive bias for inferring graphics programs. In our work, we first explore the effectiveness of LSTM-RNN versus Transformer networks as decoders for order-independent graphics programs. Since these are sequence models, we must choose an ordering of the objects in the graphics programs for likelihood training. We found that the LSTM performance was highly sensitive to the sequence ordering (random order vs. pattern-based order), while Transformer performance was roughly independent of the sequence ordering. Further, we present a policy gradient based reinforcement learning approach for better inductive bias in the decoder via multiple diverse rewards based both on the graphics program specification and the rendered image. We also explore the combination of these complementary rewards. We achieve state-of-the-art results on two graphics program generation datasets.
The large majority of computer vision work deals in the
domain of natural images or video. However, there is
tremendous potential for applying computer vision techniques to
computer-generated images, such as plots, charts,
schematics, complicated math formulas, and even a page of printed
text. For these domains, there is often a domain-specific
language for precisely specifying the image, such as
matplotlib code for a chart, PicTeX for a schematic1, and
LaTeX for math formulas and text. “De-rendering” a
computergenerated image back to the original (or a different)
domainspecific language specification can be a useful first step in
many tasks, such as changing the visual appearance of an
image
The de-rendering problem is part of a larger class of “image-to-text” problems, in which an input image is <object> <supercategory>C-1</supercategory> <category>CS-3<\category> <x-coordinate>120</x-coordinate> <y-coordinate>240</y-coordinate> <depth>1</depth> <flip>0</flip> </object> <object>....
<object>
<category>Rectangle<\category>
<x1-coordinate>7</x1-coordinate>
<y1-coordinate>1</y1-coordinate>
<x2-coordinate>11</x2-coordinate>
<y2-coordinate>16</y2-coordinate>
</object>
<object>....
mapped to some sequence of output tokens. The neural
encoder-decoder approach has proved to be very
successful for this class of problems, including image
captioning
We start our investigation with a basic image captioning
model (similar to Wu, Tenenbaum, and Kohli (2017)) and
extend it with an attention mechanism. We then swap out the
LSTM-RNN decoder with a Transformer network
Another challenge with graphics de-rendering is that
changing one or a few tokens in the specification can cause
a significant change in many pixel values (e.g., by changing
the location or color of a large object). Conversely, one can
have two images that are very close visually, yet have
completely different specifications. To this end, we explore the
error minimization in the image as well as the specification
space via a dual-modality, two-way reward reinforcement
learning approach
We empirically evaluate our methods on two image
de-rendering datasets: Noisy Shapes dataset
De-rendering a computer-generated image to a
domainspecific language provides an abstraction that is easy to
change, store, compare and match to other images. As a
consequence, there has been recent interest and work in this
area.
Recently, policy gradient-based reinforcement learning
(RL) methods have been widely used for sequence
generation tasks: machine translation
Recently,
3
Task. For each task we consider, there is a simple graphics specification language that can be used to specify a particular image. While differing in details, the overall scheme of the specifications are the same for each. A specification consists of a set of “objects”, and each object is specified by a set of properties. Examples of an object specification for each of our tasks can be seen in Fig. 1. Given an image rendered from a specification, our task is to “de-render” this image back to the original specification. We can evaluate a predicted specification by looking for exact matches between the objects in the predicted specification and the objects in the original specification. We can summarize the object matches with standard measures, such as precision, recall, F1, and intersection-over-union. While these measures describe performance on a single image, we can average these measures across a collection of images, to get a performance measure of a method overall. We provide more details in Sec. 5.2. Another approach to evaluation is to generate the image corresponding to a predicted specification, and see how well it matches the original image, using some reasonable metric on the space of images.
tions is represented by a set of objects with specific properties, our models require sequences of tokens. We convert the set of objects to a sequence of tokens via some ordering of the objects. We investigate various approaches to ordering (Sec. 7.1), and find that ordering by object type works best. Once the model predicts a sequence of tokens, we can parse it back into original structure to compute performance measures and our reward functions for reinforcement learning. 3.1
Our baseline model is similar to an image captioning model
with an attention mechanism
We will denote the convolutional features from the ResNet-18 as ffigi=1, where fi 2 Rd. For any decoder outm put token o, let Eo 2 Rd0 denote its embedding, which will be learned during training. Let st be the decoder state at step t, ot be the output token at step t, and ct be the image context vector at step t, which will be defined below. Then at step t, the decoder state st is given by st = F(ct; st 1; Eot 1 ); (1)
ResNet CNN (or)
Output Softmax
Linear Add & Norm Feed Forward Add & Norm Multi-Head Attention Add & Norm
Masked Multi-Head Attention
+
Output Embedding
Outputs (shifted right)
Positional Embedding where F is a trainable non-linear function. The context vector ct is a convex combination of the image features: ct = Pm i=1 t;ifi, where t;i are “attention weights” defined as t;i =
exp(et;i) Pm
k=1 exp(et;k) et;i = vT tanh(W fi + U st 1 + b); (2) (3) where v, W , U , and b are the trainable parameters.
Recently, there is an increasing amount of interest in
Transformer networks
The decoder of the Transformer has a stack of N
identical layers containing self-attention modules,
normalization modules, and feed-forward modules, along with
positional encoding module for output embeddings (see Fig. 2).
While the original model in
two attention mechanisms in the model: one attending to the CNN features, and another attending to different parts of the decoder state. They all have the same structure, which we describe below.
An attention mechanism in the Transformer can be viewed as a mapping from a query (Q) and a key-value (K; V ) pair to an output. An attention weight is computed using the query and key and those weights are used with values to compute the output of the attention module. Empirically, it has been proven that instead of performing a single attention function, linearly projecting the queries, keys, and values with different learned projection layers and then performing the attention function in parallel and concatenating those outputs to get the final attention module output to work better. This attention mechanism is called multi-head attention mechanism (MH), which is defined as follows: MH(Q; K; V ) = Concat(head1; ::; headh)W O headi = Attention(QWiQ; KWiK ; V WiV ) Attention(Q; K; V ) = softmax QKT dk
V (4) (5) (6) where, dk is the dimension of the queries and keys, WiQ, WiK , and WiV are the parameters of the projection matrices.
the attention sub-layers, each of the layers in the Transformer decoder contains a fully connected feed-forward network that is applied to each position of the decoder separately and identically. This network is defined as FFN(x) = max(0; xW1 + b1)W2 + b2; (7) where W1, W2, b1, and b2 are the linear projection parameters which are same across different positions but are different from layer to layer.
Positional Encoding. In the model described thus far, the model is symmetric with respect to sequence position. For example, at the bottom right of Fig. 2, the model has no structural way to determine which output embeddings come from which part of the output sequence. To remedy this issue, we concatenate a “positional encoding” (PE) to the embedding representation of the tokens. We use the sine and cosine functions for positional encoding:
PE(pos; 2i) = sin(pos=100002i=dmodel )
PE(pos; 2i + 1) = cos(pos=100002i=dmodel ) where pos is the position, i is the dimension, and dmodel is the dimension of the embedding vector representation. (8)
Traditionally, sequence generation models are trained using
a cross-entropy loss. More recently, a policy gradient-based
reinforcement learning approach has been explored for
sequence generation tasks
For this reward optimization, we use the REINFORCE
algorithm
Eos p [r(os)];
where os is the sequence of sampled tokens with ots sampled
at time step t of the decoder. We can approximate the
gradient of this loss function with respect to the parameter using
a single sample os drawn from p as:
r LRL =
(r(os)
be)r log p (os);
where the leading factor is included for variance reduction
using a baseline estimator
In this work we consider three different reward functions. Two of the rewards are based in “specification space”, which make a direct comparison between the predicted specification and the ground truth specification, and one of the rewards is based in “image space”, which compares the image rendered from the predicted specification with original input image. We also investigate using these rewards in combination, with the hope that there is complementary information in the feedback based on the two spaces.
Intersection-Over-Union Reward (IOU) As mentioned in Sec. 3, after the specification is predicted as a sequence of tokens, we can parse the sequence into a set of object specifications. The intersection-over-union (IOU) reward is based in specification space. Roughly speaking, the IOU reward gives credit for predicting objects that exactly match objects in the ground truth specification, and penalizes both Graphics Renderer
Image Distance Reward Specs from os Output Softmax Linear Ad &Norm Fe dForward Ad &Norm Multi-Head Atention Ad &Norm Masked Multi-Head Atention Ou+tput EPmosbietidonianlg Embed ing Outputs (shiftedright)
T r a n s aSm rfeo
m lep rD r e c o d e r [Object(type(boy), 60, 240), Object(type(tree), 100, 20), Object(type(bear), 360, 220)] [Object(type(boy), 60, 240), Object(type(tree), 440, 20),
Object(type(bear), 360, 220)] IOU Reward for predicting objects that do not match ground truth objects and for failing to predict objects that are part of the ground m n truth. More formally, let foigi=1 and foj gj=1 represent the objects in predicted and ground-truth specifications, respectively. Then the IOU reward is defined as: riou = count(foigim=1 \ foj gj=1)
n count(foigim=1 [ foj gjn=1) (11) The object oi in the prediction specification is the same as object oj in the ground-truth specification if and only if all the properties of these objects match exactly.
Inference Reward Our second reward, which we call the “inference reward”, is also a reward in specification space. The name is based on the “inference error”, which is a performance measure introduced in Wu, Tenenbaum, and Kohli (2017) for the Abstract Scenes dataset. While IOU is based on exact matches between predicted objects and groundtruth objects, the inference error and inference reward are based on the number of properties (within objects) that correctly match the corresponding properties in the groundtruth. For those properties specifying location in pixel coordinates, we follow Wu, Tenenbaum, and Kohli (2017) and divide the space of each coordinate into 20 bins of equal size, and we consider it a match if the predicted and groundtruth locations are in the same bin. We define the inference error as the fraction of predicted properties that fail to match the corresponding ground-truth properties. The inference reward is one minus the inference error.
Image Distance Reward Our third and final reward, the “image distance reward”, is in image space. We define it generically first, as it takes slightly different forms in our two datasets. If we let I and IR represent vectorized versions of the input image and the image rendered from the predicted specification, respectively, then we define the image distance as dimg = jjI (IR)jj22; (12) where jj jj2 is the `2-norm.
For Noisy Shapes dataset, we follow
c dimg where c is a tunable parameter.
For the Abstract Scene dataset, is a logical operator that
takes the value 0 in every position where the pixel values
“match”, and 1 in every other position. The range of possible
pixel values is 0-255 and, similarly to the discretization of
position in the inference reward, we divide the pixel value
range into 20 equisized buckets and consider pixel values to
match if they are in the same bucket. We take to be the
identity function. The image reward for the Abstract Scenes
dataset is then defined as:
rimg = 1
dimg
w h
where w and h are width and height of the image.
Joint Dual-Modality Reward Since we expect the
rewards based in specification space to be complementary to
the reward based in image space, we want a way to combine
rewards on the two spaces. One way to combine two rewards
is to create a weighted combination of individual rewards to
formulate the joint reward. Another approach is to alternate
the reward used during the learning process
3
Image-distance Reward Image-distance + IOU Reward Precision
97.6
98.5
Noisy Shapes Dataset.
Abstract Scene Dataset. The Abstract Scene dataset
Noisy Shapes Dataset. As described in the Task
description of Sec. 3, we can summarize performance on a single
image with precision, recall, F1, and IOU (intersection over
union) at the object level. Following previous work
Abstract Scene Dataset. For the abstract scene dataset,
following previous work
In all of our models, we encode the image information via
ResNet-18
6
6.1
We first compare the performance of the LSTM-RNN model (Image2LSTM+atten) to the Transformer-based model, when both are trained with cross-entropy loss. We see in PREVIOUS WORK
IMAGE2TRANSFORMER WITH REINFORCE LOSS
Recons. Avg.
Error Error CNN+LSTM (2017) NSD (full) (2017)
Table 1 that the Transformer model dominates on all measures. In particular, we highlight IOU1.0, which measures the percent of examples on which the predicted specification exactly matches the ground-truth specification. While the LSTM-RNN model achieves a 90:7% IOU1.0, the Transformer model achieves 94:1%, which is an impressive 36:5% reduction in the number of errors. We have similar performance improvements for the other metrics. We now compare the Transformer model trained with reinforcement learning, using various reward functions, to training using cross-entropy loss. Table 1 shows that, although all three reward variations have roughly the same performance, they all show significant improvement over cross-entropy training, on all measures.4 For example, the model trained with IOU reward achieved a 95:0% IOU1.0 measure, which is an impressive 15:3% reduction in the number of errors compared to the same model trained with cross-entropy loss, and a 46:2% reduction compared to the original LSTM-RNN model. Performance improvement in the other measures is at least as good. 6.2
In Table 2, we see the performance of various models on the Abstract Scene dataset, for the metrics described in Sec. 5.2. We first note that even our baseline LSTM-RNN model (Image2LSTM+atten) shows a very large error reduction compared to the results presented in Wu, Tenenbaum, and Kohli (2017) (first 4 rows of the table). This highlights the importance of an attention mechanism in these tasks. For the models trained with cross-entropy, the Transformer model shows an additional remarkable improvement over the LSTM-RNN model, across all measures.
For reinforcement learning with the Transformer model,
4The improvement of our Transformer models trained with
reinforcement learning over the corresponding cross-entropy models
is statistically significant with p < 0:01, based on the bootstrap
test
7.1
As noted above, for the Abstract Scene and the Noisy Shapes datasets that we consider, the order of the objects in the specification does not affect the final image. Nevertheless, for training both the LSTM-RNN and the Transformer models, one must choose an ordering. We ran an experiment using the Noisy Shapes dataset, in which we tried ordering the objects by shape size, shape type, and by shape position in the rendered image. We found that ordering by shape type worked best across our models, so that’s what we used for our main results in Table 1. We also wanted to investigate how important it is to have the objects in some sensible order, compared to a random ordering. Table 3 shows the results of our two models when trained with cross-entropy on specification sequences where the objects are put in random order. We find that the LSTM-RNN model performance drops dramatically (e.g. IOU1.0 drops from 90:7% to 72:0%), while the drop with Transformer networks is quite small (e.g. IOU1.0 drops from 94:1% to 93:2%.6 This is addi5For the IOU and inference reward models, this improvement is statistically significant for all metrics except reconstruction error. For the reconstruction reward model, the improvement is significant for all but the inference error metric. For the dual (IOU+Recons.) reward model, the difference is significant for all metrics (p < 0:01 for each test).
6Note that the number of model parameters is approximately the same (11.8M for Transformer model and 11.5M for LSTM model). Further, Transformer models are 2.5x faster to train in comparison to the LSTM models. During inference, both models take approximately the same time. tional evidence for Transformers being the preferred model for tasks of this type. We conduct an experiment where we vary the percentage of Noisy Shapes data used during our models’ training from 10% to 100% by steps of 20%. We observe that with less data (10%-40%), the RL-based model is approximately 2 points better (on the IOU1.0 metric) than its corresponding cross-entropy baseline. As we use more data (>60%), the gap decreases to 1 point between RL and cross-entropy models. This suggests that RL, which has the advantage of exploration, is more powerful when the data is less. Fig. 4 presents the output rendered images of the predicted specifications from Image2Transformer crossentropy model and the corresponding RL-based model with IOU+Image-distance as reward for noisy shapes dataset and IOU+Recons. as reward for abstract scene dataset. In the first example (top row in Fig. 4), the cross-entropy model predicts an extra ‘line shape’ which is not present in the ground-truth. Our RL model correctly predicts the exact same shapes present in the ground-truth. However, neither models getting the type of ‘line shape’ correct in couple of instances. In the second example (second row in Fig. 4), the cross-entropy model predicts an extra object (glasses), which is not present in the ground-truth image, and is also missing the cap on the snake. The RL model improves on the cross-entropy model by not having any extra objects, but it is also missing the cap. In the third example, both the rendered images look very similar to the ground-truth, but the crossentropy model predicts one of the objects (glasses) slightly off in position. Our RL model was able to accurately position the glasses (bottom row in Fig. 4). The better performance of RL model may be due to the image space component of the error signal, which is more sensitive to position errors, while the cross-entropy loss gives the same penalty to all incorrect positions regardless of the error size. 8
We present various neural de-rendering models based on LSTMs with attention mechanism and Transformer networks. Further, we introduce complimentary dual rewards (one in specification space and another in image space) and optimize them via reinforcement learning, and achieve stateof-the-art results. Further, our results and analyses suggest Ground-truth Images
Transformer Baseline Rendered Images Transformer RL Rendered Images that Transformers are a better choice than LSTMs for unordered sequence prediction tasks.
We thank the reviewers for their helpful comments. This work was partially supported by NSF-CAREER Award 1846185, ARO-YIP Award W911NF-18-1-0336, and a Microsoft PhD Fellowship. The views contained in this article are those of the authors and not of the funding agency.
arXiv preprint