English. In this work we obtain sentence embeddings with a recursive model using dependency graphs as network structure, trained with dictionary definitions. We compare the performance of our recursive Tree-LSTMs against other deep learning models: a recurrent version which considers a sequential connection between sentence elements, and a bag of words model which does not consider word ordering at all. We compare the approaches in an unsupervised similarity task in which general purpose embeddings should help to distinguish related content.
Italiano. In questo lavoro produciamo sentence embedding con un modello ricorsivo, utilizzando alberi di dipendenze come struttura di rete, addestrandoli su definizioni di dizionario. Confrontiamo le prestazioni dei nostri alberi-LSTM ricorsivi con altri modelli di apprendimento profondo: una rete ricorrente che considera una connessione sequenziale tra le parole della frase, e un modello bag-ofwords, che non ne considera l’ordine. La valutazione dei modelli viene effettutata su un task di similarit non supervisionata, in cui embedding di uso generale aiutano a distinguere i contenuti correlati.
Word embeddings have succeeded in obtaining
word semantics and projecting this information
in a vector space.
Despite the success of word embeddings in
capturing semantic information, they cannot obtain on
its own the composition of longer constructions,
which is essential for natural language
understanding. Thus, several methods using deep neural
networks combine word vectors for obtaining
sentence representations with linear mappings
Attending to the way the nodes of the network link with each other, two approaches are frequent in NLP: recurrent neural networks and recursive neural networks (RNN) 1. Recurrent models consider sequential links among words, while recursive models use graph-like structures for organizing the network operations. They process neighbouring words by parsing the tree order (dependency or syntactic graphs), and compute node representations for each parent recursively from the previous step until they reach the root of the tree, which gives the final sentence abstraction.
In this work, we train a variant of Tree-LSTM
models for learning concept abstractions with
dic1We use the same classification as in
The following recurrent models are capable of obtaining general purpose embeddings of sentences: Skip-thought Vectors, and DictRep.
Skip-thought Vectors
DictRep
Recurrent models have achieved good
performance results in different tasks such as polarity
detection (e.g. bidirectional LSTMs in
Despite being less explored for building
general purpose sentence embeddings, in several
classification tasks, tree-structured RNNs represent
the current state of the art. In their seminal
paper,
In
A more complete benchmark was conducted by
Learning models that build a dictionary of
embeddings have solid advantages over other supervised
approaches, since they take advantage of large
volumes of data that are already available online. The
training data of the system are pairs of
definition/target word which can be built with
dictionaries or encyclopedia descriptions (e.g. picking the
first sentences of a description as training data).
We follow previous work of
We used the Tree-LSTM as the starting point
to build our system. The input to the system
are the words conforming a definition together
with the structure of the graph with the
syntactic/dependency relationships, and the word closer
to this definition, i.e. the target. Typically the
LSTM nodes are intended for strictly sequential
information propagation. Our variant is based in
the previous work of
The main differences with the original LSTM node are the presence of two forget gates instead of one and the operation over two previous nodes of the system which modify node states and inhibitor gates. Hence, sub-indexes 1 and 2 are reserved for left and right child nodes of the graph, respectively. In this LSTM node there are no peephole connections between memory states and the inhibitor gates.
The state value in the root node is fed to the last layer of the system. Then, a non-linear transformation is applied to obtain the sentence embedding. In the basic configuration of the model, the error is measured by calculating the cosine similarity between target and predicted embeddings. The target is the embedding of the word result of the definition. Pre-trained word embeddings or random initialised embeddings might be employed. In the second case, the error is also propagated to the leaf nodes of the graph and thus the word embeddings are updated during training. We did not initialise randomly embeddings because this has consistently produced poorer results in comparison with the same model using pre-trained word embeddings.
In the network configurations of the tree-LSTM
models, we added an extra backward link between
the root node and the leaves reversing the uplink
path (as hinted in
We implemented our model with Theano
We compared DictRep (BoW and RNN) and our Tree-LSTM variant in a benchmark of unsupervised text similarity tasks and a supervised task (sentiment polarity). These tasks greatly benefit from a good representation of sentences and it requires a lot of human effort to build a dataset.
DictRep models were trained using available
data and online code. For a fair comparison, all
models employed the pre-trained word
embeddings and training data provided by
Our model employs two connection configurations: The Tree-LSTM with transformed dependency graphs and the sequential mapping of connections, which is conceptually similar to the DictRep-RNN model.
For SkipThoughts we used the code available
online (ski, ) and the pre-trained model with a
sentence representation of 4800 dimensions.
Additionally, we trained a compressed model with
sentence and word representation dimensions of 1200
and 320 respectively in about three weeks. Like in
the available model, the 80 million registers of the
BookCorpus dataset
The objective of the semantic similarity task
benchmark is to measure the similarity between a
pair of sentences. SemEval STS 2014
This is the same dataset used in previous work
For the sentiment polarity, we used as training/validation data the Sentiment Penn Treebank dataset 2. In this dataset, each sentence node is labelled with a 5-tag intensity tag from 0, the most negative, to 4. Sentences are already binarised in the same format of our TreeDict approach so that no preprocessing is needed in this task for TreeModels. We used for training and test the labels at the root node which is the the overall sentence polarity. For completeness, we repeat the analysis for a 3-label annotations over the same dataset. We used the same SVM classifier for all the models and we trained it with the sentence vectors as input. 5
The DictRep BoW model was undeniably better than the recurrent and recursive models achieving the best position in all cases (Table 1). The TreeDict-Dep model ranked second 3.
2http://nlp.stanford.edu/sentiment/treebank.html 3The character “-” indicates that some vectors for a sentence could not be obtained (e.g. due to a malformed dependency graph)
All models capture the correlations with human annotations better in neutral contexts. If there are contradictions and entailment relationships, the agreement with human annotations is less evident. Nevertheless, this behaviour is expected and also desirable, as this is an unsupervised benchmark and the system has no way of learning a similar but conflicting relationship without external help.
It is clear that BoW models offered the best performance in all the datasets. The Tree-LSTM model, which is consistently better than the sequential models, ranked second. Table 2 shows the correlation among models over the SICK similarity dataset. All the models experience strong cross-correlations between them but the TreeLSTM with dependency parsing showed the closest correlation with the BoW and recurrent models.
The Table 3 shows the performance of the models in the supervised polarity tasks. BoW and SkipThoughts models experience similar outcomes for the 5 and 3 label task. Models trained with dictionary definitions (DictRep and TreeDict) lag behind those models. However, all the networks using dependency structures have consistently beaten its sequential counterparts. This is a strong indicative of the benefits of using this more complex network structure. The difference between the different network configurations of the same model are less pronounced that in the similarity tasks but in our tests, the models that used the extra link backwards achieved small gains (at least in the 3-label task).
In previous work,
The effect of word orderings is not clear. BoW models are far from being ideal as they cannot obtain which parts are negated or the dependencies among the different elements of the sentence (e.g. the black dog chases the white cat and the black cat chases the white dog cannot be differentiated by only using BoW models).
It is important to mention that the similarity
was tested only at the root node when using
TreeLSTM. Notwithstanding, recursive models allow
to use more elaborated strategies, taking
advantage of the dependencies used to build the
relationships of the nodes in the deep network. These
strategies could combine similarities at different
levels of the sentence to obtain a more
approximate value of similarity (e.g. using a pooling
matrix with all the nodes of the parse tree
The errors during training time in held-out data were 0:57 for BoW models versus the 0:51 achieved by recurrent and recursive models. Nevertheless, better dictionary embeddings do not seem to directly translate into better performance at inferring general purpose sentence embeddings in the benchmarks. Results in the test also show that we need better mechanisms to infer sentence level representations.
Model D.BoW D.RNN T.Seq T.Dep employed. Recursive models are more expensive computationally and have a more complex implementation but they exhibit better performance in longer sentences. However, with current learning techniques recurrent and recursive models cannot offer better results than simpler models such as BoW representations of sentences in unsupervised similarity benchmarks. The results of these findings shall be confirmed in the future in more complex scenarios, such as large scale QA.
This work has been funded by the Spanish Ministerio de Economa y Competitividad through the project INRISCO (TEC2014-54335-C4-4-R). .40 .32 .31 .32 .32 .35 .35 .35 .40 .38
In this paper we introduced the use of recursive models for the generation of general purpose embeddings once they are trained by embedding dictionary definitions. We compare recurrent and recursive models in the embedding dictionary task and we test the validity of these embeddings for their use as general purpose codification of sentences with both similarity.
Results demonstrate slight advantages of the Tree recursive variant over recurrent models that learn from dictionaries, which are more frequently decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.1078.