In this work, we focus on a lightweight convolutional architecture that creates fixed-size vector embeddings of sentences. Such representations are useful for building NLP systems, including conversational agents. Our work derives from a recently proposed recursive convolutional architecture for auto-encoding text paragraphs at byte level. We propose alternations that significantly reduce training time, the number of parameters, and improve auto-encoding accuracy. Finally, we evaluate the representations created by our model on tasks from SentEval benchmark suite, and show that it can serve as a better, yet fairly low-resource alternative to popular bag-of-words embeddings. Equal contribution
Modern conversational agents often make use of retrievalbased response generation modules [Ram et al., 2018], in which the response of the agent is retrieved from a curated database. The retrieval can be implemented as similarity matching in a vector space, in which natural language sentences are represented as fixed-size vectors. Cosine and Euclidean distances typically serve as similarity measures. Such approaches have been applied by participants of recent chatbot contests: The 2017 Alexa Prize [Pichl et al., 2018; Liu et al., 2017; Serban et al., 2017], and The 2017 NIPS Conversational Intelligence Challenge [Chorowski et al., 2018; Yusupov and Kuratov, 2017]. Retrieval-based modules are fast and predictable. Most importantly, they enable soft matching between representations. Apart from this straightforward application in dialogue systems, sentence embeddings are applicable in downstream NLP tasks relevant to dialogue systems. Those include sentiment analysis [Pang and Lee, 2008], question answering [Weissenborn et al., 2017], censorship [Chorowski et al., 2018], or intent detection.
Due to the temporal nature of natural languages, recurrent neural networks gained popularity in NLP tasks. Active research of different architectures led to great advances and eventually a shift towards methods using the transformer architecture [Vaswani et al., 2017] or convolutional layers [Bai et al., 2018; van den Oord et al., 2017a; van den Oord et al., 2017b] among researchers and practitioners alike. In this work, we focus on a lightweight convolutional architecture that creates fixed-size representations of sentences.
Convolutional neural networks have the inherent ability to detect local structures in the data. In the context of conversational systems, their speed and memory efficiency eases deployment on mobile devices, allowing fast response retrieval and better user experience. We analyze and build on the recently proposed Byte-Level Recursive Convolutional Auto-Encoder (BRCA) for text paragraphs [Zhang and LeCun, 2018], which is able to auto-encode text paragraphs into fixed-size vectors, reading in bytes with no additional preprocessing.
Based on our analysis, we are able to explain the behavior of this model, point out possible enhancements, achieve auto-encoding accuracy improvements and an order of magnitude training speed-up, while cutting down the number of parameters by over 70%. We introduce a balanced padding scheme for input sequences and show that it significantly improves convergence and capacity of the model. As we find byte-level encoding unsuitable for embedding sentences, we demonstrate its applicability in processing sentences at wordlevel. We train the encoder with supervision on Stanford Natural Language Inference corpus [Bowman et al., 2015; Conneau et al., 2017] and investigate its performance on various transfer tasks to assess quality of produced embeddings.
The paper is structured as follows: in Section 2 we introduce some of the notions that appear in the paper. Section 3 discusses relevant work for sentence vector representations. Details of the architecture can be found in Section 4. Section 5 presents the analysis of the auto-encoder and motivations for our improvements. Section 6 demonstrates supervised training of word-level sentence encoder, and evaluates it on tasks relevant to conversational systems. Section 7 concludes the paper. 2
An information retrieval conversational agent selects a response from a fixed set. Let D = f(ik; rk)gk be a set of conversational input-response pairs, and q be a current user’s input. Two simple ways of retrieving a response rk from available data are [Ritter et al., 2011; Chorowski et al., 2018]: return rk most similar to user’s input q, return rk for which ik is most similar to q.
Utterances ik; rk; q may be represented (embedded) as realvalued vectors.
Many NLP systems represent words as points in a continuous vector space using word embedding methods [Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2016]. They are calculated based on co-occurrence of words in large corpora. The same methods were applied to obtain sentence embeddings only with partial success, due to the combinatorial explosion of all possible word combinations, which make up a sentence. Instead, Recurrent Neural Networks (RNNs), autoregressive models that can process input sequences of an arbitrary length, are thought to be a good method for handling variable-length textual data, with Long Short-Term Memory network (LSTM) [Hochreiter and Schmidhuber, 1997] being the prime example of such.
Recently, RNNs have been reported to be successfully replaced by convolutional architectures [Bai et al., 2018; van den Oord et al., 2017a; van den Oord et al., 2017b]. Convolutional neural networks are traditionally associated with computer vision and image processing [Krizhevsky et al., 2012; Redmon et al., 2015]. They primarily consist of convolutional layers that apply multiple convolutions to the input, followed by pooling layers that are used for reducing the dimensionality of the hidden state. Convolutional networks are efficient during training and inference: they utilize few parameters and do not require sequential computations, making hardware parallelism easy to use. Due to their popularity in image processing, there are efficient implementations that scale well.
Residual connection [He et al., 2015] is a connection that adds an unchanged input to the output of the layer or block of layers. During the forward pass it provides upper layers with undistorted signal from the input and intermediate layers. During the backward pass it mitigates vanishing and exploding gradient problems [Hochreiter et al., 2001].
Batch Normalization [Ioffe and Szegedy, 2015] (BN) applies normalization to all activations in every minibatch. Typical operations used in neural networks are sensitive to changes in the range and magnitude of inputs. During training the inputs to upper layers vary greatly due to changes in weights of the model. Normalization of the signal in each layer has the potential to alleviate this problem. Both BN and residual connections enable faster convergence by helping with forward and backward flow of information. They were also crucial in training our models. 3
There are many methods for creating sentence embeddings, the simplest being averaging word-embedding vectors of a given sentence [Joulin et al., 2016; Le and Mikolov, 2014]. SkipThought [Kiros et al., 2015] generalizes idea of unsupervised learning of word2vec word embeddings [Mikolov et al., 2013]. It is implemented in the encoder-decoder setting using LSTM networks. Given a triplet of consecutive sentences (si 1; si; si+1), the encoder creates a fixed-size embedding vector of the sentence si, and the decoder tries to x t R sPo E D ive
s ECO rceuR D xe r P x t R sPo ED isve O rcu C eR EN rxe
P
Feature Vector
xN-1
Applied K-r times (variable) x N x N x N
Applied K-r times (variable) x N x N 2K input tokens (variable length)
tsxo Conv & BN x N P irsveu Conv & BN xNA-p1plied ceR Expand Conv & BN K-r times e v i rs u c e R x e r P
Feature Vector
Applied
K-r times x N x N 2K input tokens ( xed length) Output byte probabilities Output byte probabilities
(a) BRCA (b) Our model generate sentences si 1 and si+1 from this representation. SkipThought vectors have been shown to preserve syntactic and semantic properties [Kiros et al., 2015], so that their similarity is represented in the embedding space.
InferSent model [Conneau et al., 2017] shows that training embedding systems with supervision on a natural language inference task may be superior to an unsupervised training. Recently, better results were obtained by combining supervised learning on an auxiliary natural language inference corpus with learning to predict the correct response on a conversational data corpus [Yang et al., 2018]. 4
Our model builds on the Byte-Level Recursive Convolutional Auto-Encoder [Zhang and LeCun, 2018] (BRCA). Both models use a symmetrical encoder and a decoder. They encode a variable-length input sequence as a fixed-size latent representation, by applying a variable number of convolve-and-pool stages (Figure 1). Unlike in autoregressive models, the recursion is not applied over the input sequence length, but over the depth of the model. As a result, each recursive step processes all sequence elements in parallel. 4.1
Encoder Our encoder uses two operation groups from the BRCA: a prefix group and a recursive group. The prefix group consists of N temporal convolutional layers, and the recursive group consists of N temporal convolutional layers followed by a max-pool layer with kernel size 2. For each sequence, the prefix group is applied only once, while the recursive group is applied multiple times, sharing weights between all applications. All convolutional layers have d = 256 channels, kernels of size 3 and are organized into residual blocks, with 2 layers per block, ReLU activations, residual connections, and Batch Normalization (see Section 5.5 for details).
The encoder of our model reads in text, by sentence or by paragraph, as a sequence of discrete tokens (e.g. bytes, characters, words). Each input token is embedded as a fixedlength d-dimensional vector. Unlike [Zhang and LeCun, 2018], where input sequence is zero-padded to the nearest power of 2, we pad the input to a fixed length 2K , distributing the characters evenly across the input. We motivate this decision by the finding that the model zero-padded to the nearest power of 2 does not generalize to sentences longer than those seen in the training data (see Section 5.2).
First, the prefix group is applied, which retains the dimensionality and the number of channels, i.e., sequence length and embedding size. Let d 2r be the dimensionality of the latent code output by the encoder. The encoder then applies the recursive group K r times. Note that with r one may control size of a latent vector (the level of the compression). With the max-pooling, every application halves the length of the sequence. Weights are shared between applications. Finally, the encoder outputs a latent code of size d 2r. Unlike [Zhang and LeCun, 2018], we do not apply any linear layers after recursions. Our experiments have shown that they slightly degrade the performance of the model, and constitute the majority of its parameters. 4.2
Decoder The decoder acts in reverse. First, it applies a recursive group consisting of a convolutional layer which doubles the number of channels to 2d, which is followed by an expand transformation [Zhang and LeCun, 2018] and N 1 convolutional layers. Then it applies a postfix group of N temporal convolutional layers. Similarly to the encoder, the layers are organized into residual blocks with ReLU activations, Batch Normalization, and have the same dimensionality and kernel size. We double the size of input in the residual connection, which bypasses the first two convolutions of the recursive group, by stacking it with itself. We found it crucial for convergence speed to use only residual blocks in the network, also in the expand block.
The decoder applies its recursive group K r times. Each application doubles the numbers of channels, while the expand transformation reorders and reshapes the data to effectively double the length of the sequence while the number of channels is unchanged and equals d. The postfix group processes a tensor of size 2K d and retains its dimensionality. The output is interpreted as 2K probability distributions over possible output elements. Adding an output embedding layer, either tied with input embedding layer or separate, slowed down training and did not improve the results. At the end, a Softmax layer is used to compute output probabilities over possible bytes. Note that output probabilities are independent from one another conditioned on the input.
In this section we justify our design choices through a series of experiments with the BRCA model and report their outcomes.1 5.1
Data In order to produce comparable results, we prepared an English Wikipedia dataset with similar sentence length distribution to [Zhang and LeCun, 2018]. Namely, we took at random 11 million sentences from an English Wikipedia dump extracted with WikiExtractor2, so that their length distribution would roughly match that of [Zhang and LeCun, 2018] (Table 1). In experiments with random data, we generate random strings of a-zA-Z0-9 ASCII characters. Natural language texts are highly compressible due to their low entropy, which results from redundancy of the language [Levitin and Reingold, 1994]. In spite of this, the considered models struggle to auto-encode 1024-byte short paragraphs into 1024-float latent vectors, which are 4096-byte given their sheer information content. Transition from discrete to continuous representation and inherent inefficiency of the model are likely to account for some of this overhead.
One can imagine an initialization of weights that, given the over-capacitated latent representation, would make the network perform identity for paragraphs up to 128 bytes long3.
We confirmed those speculations experimentally, training models on paragraphs of random printable ASCII characters, namely random strings of a-zA-Z0-9 symbols (Table 2).
The empirical capacity of our model is 128 bytes, which sheds light on the amount of overhead. This model has to be trained on paragraphs longer than 512 bytes in order to learn useful, compressing behavior given a 1024-float latent representation. Both models were trained on sentences of length up to 128 bytes and evaluated on unseen data. The LSTM model did not perfectly learn the identity function, even though it was solving an easier task of predicting the character given the correct prefix. However, the LSTM model generalized much better on longer sequences, where performance of BRCA deteriorated rapidly (Table 3). 5.4
Balanced Padding of Input Sequences We found BRCA difficult to train. The default hyperparameters given by the authors [Zhang and LeCun, 2018] are singlesample batches, SGD with momentum 0:9, and a small learning rate 0:01 with 100 epochs of training. In our preliminary experiments, increasing the batch size by batching paragraphs of the same length improved convergence on datasets with short sentences (mostly up to 256 bytes long), but otherwise deteriorated on the Wikipedia dataset, where roughly 50% paragraphs are longer than 256 bytes. We suspect that the difficulty lies in the difference of the underlying tasks: long paragraphs require compressive behavior, while short ones merely require learning the identity function. Updating network parameters towards one tasks hinders the performance on the others, hence the necessity for careful training.
In order to blend in both tasks, we opted for padding input sequences into fixed-length vectors. We find it sensible to fix maximum length of input sentence, since the model does not generalize to unseen lengths anyway. Variable length of input in BRCA does save computations, however we found fixing input size to greatly improve training time, despite the overhead.
In order to make the tasks more similar, we propose balanced padding of the inputs (Figure 2). Instead of padding from the right up to 2K bytes, we pad to the nearest power of
Tom tenw oehm
Tom tenw oehm (a) Unbalanced padding to se- (b) Unbalanced padding to sequence of length 8 quence length 16 m o T t n e w e m o h m o T t n e w e m o h (c) Balanced padding to se- (d) Balanced padding to sequence of length 8 quence length 16 2 and distribute the remaining padding equally in between the bytes. We hypothesized that it could free convolutional layers from the burden of propagating the signal from left to right in order to fill the whole latent vector, as it would be the case, e.g., when processing a 64-byte paragraph padded with 960 empty tokens from the right to form 1024-byte input. Empirically, this trades additional computations for better convergence characteristics. 5.5
Batch Normalization Fixed-length, balance-padded inputs allow easy mixing of paragraphs of different lengths within a batch, in consequence allowing raising the batch size, applying Batch Normalization and raising the learning rate. This enables a significant speedup in convergence and better auto-encoding accuracy (see Section 5.6). However, the statistics collected by BN layers differ during each of the K r recursive steps, even though the weights of convolutions in the recursive layers are shared.
This breaks auto-encoding during inference time, when BN layers have fixed mean and standard deviation collected over a large dataset. We propose to alleviate this issue by either: a) collecting separate statistics for each recursive application and each input length separately, or b) placing a paragraph inside a batch of data drawn from the training corpus during inference and calculating the mean and the standard deviation on this batch. We also experimented with the instance normalization [Ulyanov et al., 2016], which performs the normalization of features of a single instances, rather than of a whole minibatch. We have found that the instance normalization improved greatly upon the baseline model with no normalization, but performed worse than batch normalization.
BRCA has been introduced with linear layers in the postfix/prefix groups of the encoder/decoder. In our experiments, removing those layers from the vanilla BRCA lowered accuracy by a few percentage points. Conversely, our model benefits from not having linear layers. We observed faster convergence and better accuracy without them, while reducing the number of parameters from 23:4 million to 6:67 million. 5.6
Auto-Encoding Performance Our training setup is comparable with that of BRCA [Zhang and LeCun, 2018]. In each epoch, we randomly select 1 million sentences from the training corpus. We trained using SGD with momentum 0:5 in batches of 32 paragraphs of random length, balanced padded to 2r = 1024 tokens, including (a) Input sentence: One of Lem’s major recurring themes, beginning from his very first novel, ”The Man from Mars” (. . . ) (b) Input sentence: Typical fuel is denatured alcohol, methanol, or isopropanol (. . . ) a special end-of-sequence token. The training was run for 16 epochs, and learning rate was multiplied by 0:1 every epoch after the 10th epoch. The model suffered from the exploding gradient problem [Hochreiter et al., 2001], and gradient clipping stabilized the training, enabling even higher learning rates. With clipping, we were able to set the learning rate as high as 30:0, cutting down training time to as low as 5 epochs.
Figure 4 shows auto-encoding error rate on the test set by sentence length. Our best model achieved 1:5% test error, computed as average byte-level error on the English Wikipedia dataset.
Finally, we were able to train a static version of our model (i.e., with no shared weights in the recursion group) in comparable time, closing a huge gap in convergence of recursive and static models in vanilla BRCA. 5.7
Generalization We investigated which inputs influence correct predictions of the network using the method of Integrated Gradients [Sundararajan et al., 2017]. We have produced two heatmaps of input-output relationships for short (128 bytes) and long (1024 bytes) paragraphs in our best model (Figure 3). In theory, a model performing identity should have a diagonal heatmap. Our model finds relations within bytes of individual words, rarely crossing word and phrases boundaries. In this sense, it fails to exploit the ordering of words. However, 18% 16% 14% r rro12% e iton10% c u trsn 8% o ceR6% 4%
Ours (clipping, BN, 5 epochs)
Ours (BN, 16 epochs) Ours(static, BN, 16 epochs)
Ours(IN, 16 epochs)
BRCA 2% 0% 0.020.030.020.080.05 0.030.040.020.230.44 0.42 65-128 B 129-256 B
Paragraph length 1.02 0.61 257-512 B the order is mostly preserved in the latent vector.
Early in the training the model learns to output only spaces, which are the most common bytes in an average Wikipedia paragraph. Later during training, it learns to correctly rewrite spaces, while filling in the words with vowels, which are the most frequent non-space characters. Interestingly, the compressing behavior seems to be language-specific and triggered only by longer sequences. Figure 5 presents input sentences and auto-encoded outputs of our best model, trained on English Wikipedia, for English, French and random input sequences. 6
Following the methods and work of [Conneau et al., 2017], we apply our architecture to a practical task. Namely, we train models consisting of the recursive convolutional word-level encoder and a simple three-layer fully-connected classifier on Stanford Natural Language Inference (SNLI) corpus [Bowman et al., 2015]. This dataset contains 570k sentence pairs, each one described by one of three relation labels: entailment, contradiction, and neutral. Then we test encoders on various transfer tasks measuring semantic similarities between sentences.
The encoder of each model has a similar architecture to the previously described byte-level encoder. However, instead of bytes it takes words as its input sequence. Our best encoder has N = 8 layers in each group. The recursive group is applied K times where 2K is length of a padded input sequence, so that the latent vector is of the size of a word vector. We use pre-trained GloVe vectors4 and we do not fine-tune them. We compared both fixed-length balanced, and variable length input paddings. In fixed-length padding, up to first 64 words are taken from each sentence. We also compare ensemble of our best trained model and bag-of-words as a sentence representation. Let v be the output vector of the encoder, and u = m1 Pim=1 e(wi) be the average of word vectors of the 4https://nlp.stanford.edu/projects/glove/ sentence, where m is the length of the sentence, wi is its i-th word, and e(w) is the GloVe embedding of the word w. Final embedding is the sum x = v + u.
Table 4 presents results for word-level recursive convolutional encoder (WRCE), word-level model with fixed balanced padding (Ours), and an ensemble of our model and an average embedding of the input sequence (Ours + BoW). We compare them with a baseline model (BoW - average of GloVe vectors for words in a sentence) on SNLI and other classification tasks, SICK-Relatedness [Marelli et al., 2014], and STSf12-16g tasks. The SentEval5 tool was used for these experiments.
For certain tasks, especially those measuring textual similarity, which are useful in retrieval-based response generation in dialogue systems, presented models perform better than bag-of-words. However, they are still not on par with LSTMbased methods [Conneau et al., 2017; Kiros et al., 2015] that generate more robust embeddings. LSTM models are autoregressive and thus require slow sequential computations. They are also larger, with the InferSent model [Conneau et al., 2017] having over 30 times more parameters than convolutional encoders presented in this section. In addition, our architecture can share word embedding matrices with other components of a conversational system, since word embeddings are ubiquitous in different modules of NLP systems.
In order to qualitatively assess how the results for those
tasks transfer to the actual dialogue system, we have
compared some retrieved responses of a simple retrieval-based
5https://github.com/facebookresearch/
SentEval
agent, which matches user utterance with a single quote from
Wikiquotes [Chorowski et al., 2018]. We present a
comparison of our word-level sentence encoder with the bag-of-word
method in response retrieval task (Figure 6). Human
utterances from the training data of NIPS 2017 Conversational
Challenge6 have been selected as input utterances. We match
them with the closest quote from Wikiquotes, using a method
similar to the one used in Poetwannabe chatbot [Chorowski
et al., 2018]. All utterances have been filtered for foul speech
The problem of efficiently producing good and robust sentence embeddings seems far from being solved. In this work, which we consider a step forward in exploration of possible tools and solutions, we analyzed and proposed improvements to the BRCA model by Xiang Zhang and Yann LeCun [2018]. With extensive usage of convolutions, our model is efficient in terms of computation and memory usage. By analyzing BRCA we were able to diagnose problems with its training, such as exploding gradients, and understand the difficulty in auto-encoding of long paragraphs, especially in the initial stage of training. Furthermore, we showed how to 6http://convai.io/2017/data/ 7https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/tokenizer/ python-tokenizer/moses.py successfully apply batch normalization with recursive layers and investigate input-output relations with Integrated Gradients method.
The recursive convolutional architecture benefits from the ease of training and low number of parameters. Due to our realization that in the current byte-level setting, input-output relations rarely cross word boundaries, we demonstrate applicability of the architecture in a word-level setting as a sentence embedder. Furthermore, a good performance on semantic similarity tasks while using little resources demonstrates it practical usefulness for dialog systems.
The authors thank Paweł Rychlikowski and Michał Zapotoczny for fruitful discussions, and Xiang Zhang for help towards a better understanding of the original model. The research was partially supported by National Science Center (Poland) grant Sonata 8 2014/15/D/ST6/04402.