Modern methods of natural language processing (NLP) are based on complex neural network architectures, where language units are represented in a metric space [23,28,29,9,30]. Such a phenomenon allows us to express linguistic features (i.e., morphological, lexical, syntactic) mathematically.
The method of obtaining such representation and their interpretations were described in multiple overview works. Almeida and Xexéo surveyed different types of static word embeddings [1], and Liu et al. [18] focused on contextual representations found in the most recent neural models. Belinkov and Glass [4] surveyed the strategies of interpreting latent representation. Best to our knowledge, we are the first to focus on the syntactic and morphological abilities of the word representations. We also cover the latest approaches, which go beyond the interpretation of latent vectors and analyze the attentions present in state-of-theart Transformer models.
obtained by counting word frequency across documents on distinct subjects.
In more recent approaches, a shallow neural network is used to predict each word based on context (Word2Vec [23]) or approximate the frequency of coocurence for a pair of words (GloVe [28]). One explanation of the effectiveness of these algorithms is the distributional hypothesis [11]: "words that occur in the same contexts tend to have similar meanings".
The main disadvantage of the static word embeddings is that they do not take into account the context of words. This is especially an issue for languages rich in words that have multiple meanings.
The contextual embeddings introduced in [29] and [22] are able to encode both words and their contexts. They are based on recurrent neural networks (RNN) and are typically trained on language modeling or machine translation tasks using large text corpora. The outputs of the RNN layers are context-dependent representations that are proven to perform well when used as inputs for other NLP tasks with much less training data available.
Another improvement of context modeling was possible thanks to the attention mechanism [2]. It allowed passing the information from the most relevant part of the RNN encoder, instead of using only the contextual representation of the last token.
The most recent and widely used architecture is the Transformer [32]. It consists of several (6 to 24) layers, and each token position in each layer has the ability to attend to any position in the previous layer using a self-attention mechanism. Training such architecture can be easily parallelized since individual tokens can be processed independently; their positions are encoded within the input embeddings. An example of visualization of attention distribution computed in Transformer trained for language modeling (BERT [9]) is presented in Figure 1.
In addition to vectors, Transformer includes latent representation in the form of self-attention weights, which are two-dimensional matrices. We summarize the research on the syntactic properties of attention weights in Section 5.
This sections describes the metrics used to evaluate syntactic information captured by the word embeddings and latent representation.
In the recent revival of word embeddings [23,28], a strong focus was put on examining the phenomenon of encoding analogies in multidimensional space. That is to say, the shift vector between pairs of analogous words is approximately constant, e.g., the pairs drinking -drank, swimming -swam in Figure 2.
Syntactic analogies of this type are particularly relevant for this overview. They include the following relations: adjective -adverb; singular -plural; adjective -comparative -superlative; verb -present participle -past participle. The syntactic analogy is usually evaluated on Google Analogy Test Set [23]. 1An evaluation example consists of two word pairs represented by the embeddings: (v 1 , v 2 ), (u 1 , u 2 ). We compute the analogy shift vector as the difference between embeddings of the first pair s = v 2 − v 1 . The result is positive if the nearest word embedding to the vector u 1 + s is u 2 .
Sequence tagging is a multiclass classification problem.
The aim is to predict the correct tag for each token of a sequence. A typical example is the part of speech (POS) tagging. The accuracy evaluation is straightforward: the number of correctly assigned tags is divided by the number of tokens.
The inference of reasonable syntactic structures from word representations is the most challenging task covered in our survey. There are attempts to predict both the dependency [12,31,15,7] and constituency trees [21,13]. Dependency trees are evaluated using unlabeled attachment score (UAS) or its undirected variant (UUAS):
The equation for Labeled Attachment Score is the same, but it requires predicting a dependency label for each edge.
For constituency, trees we define precision (P) and recall (R) for correctly predicted phrases.
Usually, F1 is reported, which is a harmonic mean of precision and recall.
In Section 5 we describe the examination of syntactic properties of self-attention matrices. It can be evaluated using Dependency Alignment [34] which sums the attention weights at the positions corresponding to the pairs of tokens forming a dependency edge in the tree.
Dependency Accuracy [35,7,15] is an alternative metric; for each dependency label it measures how often the relation's governor/dependent is the most attended token by the dependent/governor.
Notation: E is a set of all dependency tree edges and E l,d is a subset of the edges with the label l and with direction d, i.e., in dependent to governor direction the first element of the tuple i is dependent of the relation and the second element j is the governor; A is a self-attention matrix and A i,• denotes i th row of the matrix; N is the sequence length.
In this section, we summarize the research on the syntactic information captured by vector representations of words.
We devote a significant attention to POS tagging, which is a popular evaluation objective. Even though it is a morphological task, it is highly relevant to syntactic analysis.
The first wave of research on the vector representation of words focused on the statistical distribution of words across distinct topics -Latent Semantic Analysis [8]. It captured statistical properties of words, yet there were no positive results in syntactic analogies retrieval nor encoding syntax. Google Analogy Test Set was released together with a popular word embedding algorithm Word2Vec [23]. One of the exceptional properties of this method was its high accuracy in the analogy tasks. In particular, the best configuration found the correct syntactic analogy in 68.9 % of cases.
The GloVe embeddings improved the results on syntactic analogies to 69.3% [28]. Much more significant improvement was reported for semantic analogies. They also outperform the variety of other vectorization methods.
In [24] a simple recurrent neural network was trained by language modeling objective. The word representation is taken from the input layer. The evaluation from [23] shows that Word2Vec performs better in syntactic analogy task. This observation is surprising because representations from RNN were proven effective in transfer to other syntactic tasks (we elaborate on that in Sections 4.2 and 4.3). We think that possible explanations could be: 1. the techniques of RNN training have crucially improved in recent years; 2. syntactic analogy focuses on particular words, while for other syntactic tasks, the context is more important.
Measuring to what extent a linguistic feature such as POS is captured in word representations is usually performed by the method called probing. In probing, the parameters of the pretrained network are fixed, the output word representations are computed as in the inference mode and then fed to a simple neural layer. Only this simple layer is optimized for a new task.
The number of probing experiments rose with the advent of multilayer2 RNNs trained for language modeling and machine translation.
Belinkov et al. [3] probe a recurrent neural machine translation (NMT) system with four layers to predict part of speech tags (along with morphological features). They use Arabic, Hebrew, French, German, and Czech to English pairs. They observe that adding a character-based representation computed by a convolutional neural network in addition to word-embedding input is beneficial, especially for morphologically rich languages.
In a subsequent study [4], the source language of translation now is English and the experiments are conducted solely for this language. It is noted that the most morphosyntactic representation is usually obtained in the middle layers of the network.
The influence of using a particular objective in pretraining RNN model is comprehensively analyzed by Blevins et al. [5]. They pre-train models on four objectives: syntactic parsing, semantic role labeling, machine translation, and language modeling. The two former objectives may reveal morphosyntactic information to a larger extent than other mentioned here settings. Particularly, the probe of RNN syntactic parser achieves near-perfect accuracy in part of speech tagging.
The introduction of ELMo [29] brought a remarkable advancement in transfer learning from the RNN language model to a variety of other NLP tasks. The authors examined POS capabilities of the representations and compared the results with the neural machine translation system CoVe [22], which also uses RNN architecture.
Zhang et al. [39] perform further experiments with CoVe and ELMo. They demonstrate that language modeling systems are better suited to capture morphology and syntax in the hidden states than machine translation, if comparable amounts of data are used to train both systems. Moreover, the corpora for language modeling are typically more extensive than for machine translation, which can further improve the results.
Another comprehensive evaluation of morphological and syntactic capabilities of language models was conducted by Liu et al. [17]. Probing was applied to a language model based on the Transformer architecture (BERT) and compared with ELMo and static word embeddings (Word2Vec). They observe that the hidden states of Transformer do not demonstrate a major increase in probed POS accuracy over the RNN model, even though it is more complex and consists of a larger number of parameters.
POS tag probing was also performed for languages other than English. For instance, Musil [25] trains translation systems (with RNN and Transformer architecture) from Czech to English and examines the learned input embeddings of the model and compares them to a Word2Vec model trained on Czech. In Figures 3 and 4, we present a comparison of different settings for POS tag probing. Each point denotes a pair of results obtained in the same paper and the same dataset, but with different types of embeddings or pretraining objectives. Therefore, we can observe that the setting plotted on the y-axis is better than the x-axis setting if the points are above identity function (red dashed line). We cannot say whether a method represented by another point performs better, as the evaluation settings differ.
Figure 4 clearly shows that the RNN contextualization helps in part of speech tagging. As expected, the information about neighboring tokens is essential to predict morphosyntactic functions of words correctly. It is especially true for the homographs, which can have various part of speech in different places in the text.
The influence of RNN's pre-training task is presented in Figure 3. Machine translation captures much better POS information than auto-encoders, which can be interpreted as translation from and to the same language. It is likely that the latter task is straightforward and therefore does not require to encode morphosyntax in the latent space. The difference between the results of machine translation and language modeling is small. Zhang et al. [39] show that using a larger corpus for pre-training improves the POS accuracy. The main advantage of language models is that monolingual data is much easier to obtain than parallel sentences necessary to train a machine translation system.
Extraction of dependency structure is more demanding because instead of prediction for single tokens, every pair of words need to be evaluated.
Blevins et al. [5] propose a feed-forward layer on top of a frozen RNN representation to predict whether a dependency tree edge connects a pair of tokens. They concatenate the vector representation of each of the words and their element-wise product. Such a representation is fed as an input to the binary classifier. It only looks on a pair of tokens at a time, therefore predicted edges may not form a valid tree.
Another approach, induction of the whole syntactic structures from latent representations was proposed by Hewitt and Manning [12]. Their syntactic probing is based on training a matrix which is used to transform the output of network's layers (they use BERT and ELMo). The objective of the probing is to approximate dependency tree distances between tokens 3 by the L2 norm of the difference of the transformed vectors. Probing produces the approximate syntactic pairwise distances for each pair of tokens. The minimum spanning tree algorithm is used on the distance matrix to find the undirected dependency tree. The best configuration employs the 15th layer of BERT large and induces treebank with 82.5% UAS on Penn Treebank with Stanford Dependency annotation (relation directions and punctuation were disregarded in the experiments). The result for BERT is significantly higher than for ELMo, which gave 77.0% when the first layer was probed.
The paper also describes an alternative method of approximating the syntactic depth by the L2 norm of latent vector multiplied by a trainable matrix. The estimated depths allow prediction of the root of a sentence with 90.1% accuracy when representation from the 16th layer of BERT large is probed.
The subsequent paper by Chi et al. [6] applies the setting from [12] to the multilingual language model mBERT. They train syntactic distance probes on 11 languages and compare UAS of induced trees in four scenarios: 1. training and evaluating on the same languages; 2. training on a single language, evaluating on a different one; 3. training on all languages except the evaluation one; 4. training on all languages, including the evaluation one. They demonstrate that the transfer is effective as the results in all the configurations outperform the baselines 4 . Even in the hardest case -zero-shot transfer from just one language, the result is at least 6.9 percent points above the baselines (for Chinese). Nevertheless, for all the languages, no transfer-learning setting can beat the training and evaluating a probe on the same language.
The paper includes analysis of intrinsic features of the BERT's vectors transformed by a probe. Noticeably, the vector differences between the representations of words connected by dependency relation are clustered by relation labels, see figure 5.
Multilingual BERT embeddings are also analyzed by Wang et al. [36]. They show that even for the multilingual vectors, the results can be improved by projecting vector spaces across languages. They use Biaffine Graph-based Parser by Dozat and Manning [10], which consists of multiple RNN layers. Therefore, the experiment is not strictly comparable with probing as the most of syntactic information is captured by the parser, and not by the embeddings. The article compares different types of vector representations fed as an input to the parser. It is demonstrated that cross-lingual transformation on mBERT embedding improves the results significantly in LAS of parser trained on English and evaluated on 14 languages (including English); on average, from 60.53% to 63.54%. In comparison to other cross-lingual representations, the proposed method outperforms transformed static embeddings (Fast-Text with SVD) and also slightly outperforms contextual embeddings (XLM).
Besides the vector representations of individual tokens, the Transformer architecture offers another representation 4 There are two baselines: right-branching tree and probing on randomly initialized mBERT without pretraining Figure 5: Two dimensional t-SNE visualization of probed mBERT embeddings from [6]. Analysis of the clusters shows that embeddings encode information about the type of dependency relations and, to a lesser extent, language. with a possible syntactic interpretation -the weights of the self-attention heads. In each head, information can flow from each token to any other one. These connections may be easily analyzed and compared to syntactic relations proposed by linguists. In this section, we will summarize different approaches of extracting syntax from attention. We present the methods both for dependency and constituency structures.
Raganato and Tiedemann [31] induce dependency trees from self-attention matrices of a neural machine translation encoder. They use the maximum spanning tree algorithm to connect pairs of tokens with high attention. Gold root information is used to find the direction of the edges. Trees extracted in this way are generally worse than the right-branching baseline (35.08 % UAS on PUD) and outperform it slightly in a few heads. The maximum UAS is obtained when a dependency structure is induced from one head of the 5th layer of English to Chinese encoder -38.87% UAS. Nevertheless, their approach assumes that the whole syntactic tree may be induced from just one attention head.
Recent articles focused on the analysis of features and classification of Transformer's self-attention heads. Vig and Belinkov [34] apply multiple metrics to examine properties of attention matrices computed in a unidirectional language model (GPT-2 [30]). They showed that in some heads, the attentions concentrate on tokens representing specific POS tags and the pairs of tokens are more often attended one to another if an edge in the dependency tree [35] also observed alignment with dependency relations in the encoders of neural machine translation systems from English to Russian, German, or French. They have evaluated dependency accuracy for four dependency labels: noun subject, direct object, adjective modifier, and adverbial modifier. They separately address the cases where a verb attends to a dependent subject, and subject attends to governor verb. The heads with more than 10% improvement over a positional baseline are identified as syntactic 6 . Such heads are found in all encoder layers except the first one. In further experiments, the authors propose the algorithm to prune the heads from the model with a minimal decrease in translation performance. During pruning, the share of syntactic heads rises from 17% in the original model to 40% when 75% heads are cut out, while a change in translation score is negligible. These results support the claim that the model's ability to capture syntax is essential to its performance in non-syntactic tasks.
A similar evaluation of dependency accuracy for the BERT language model was conducted by Clark et al. [7].
They identify syntactic heads that significantly outperform positional baseline for the following labels: prepositional object, determiner, direct object, possession modifier, auxiliary passive, clausal component, marker, phrasal verb particle. The syntactic heads are found in the middle layers (4th to 8th). However, there is no single head that would capture the information for all the relations.
In another experiment, Clark et al. [7] induce a dependency tree from attentions. Instead of extracting structure from each head [31] they use probing to find the weighted average of all heads. The maximum spanning tree algorithm is used to induce the dependency structure from the average. This approach produces trees with 61% UAS and can be improved to 77% by making weights dependent on the static word representation (fixed GloVe vectors). Both the numbers are significantly higher than right branching baseline 27%.
A related analysis for English (BERT) and the multilingual variant (mBERT) was conducted by Limisiewicz et al. [15]. We have observed that the information about one dependency type is split across many self-attention heads and in other cases, the opposite happens -many heads have the same syntactic function. They extract labeled dependency trees from the averaged heads and achieves 52% UAS and show that in the multilingual model (mBERT) specific relation (noun subject, determines) are found in the same heads across typologically similar languages.
There are fewer papers devoted to deriving constituency syntax tree structures.
Mareček and Rosa [21] examined the encoder of the machine translation system for translation between English, French, and German. We observed that in some heads, stretches of words attend to the same token forming shapes similar to balustrades (Figure 7). Furthermore, those stretches usually overlap with syntactic phrases. This notion is employed in the new method for constituency tree induction. In their algorithm, the weights for each stretch of tokens are computed by summing the attention focused on the balustrades and then inducing a constituency tree with CKY algorithm [26]. As a result, we produce trees that achieve up to 32.8% F1 score for English sentences, 43.6% for German and 44.2% for French. 7 The results can be improved by selecting syntactic heads and using only them in the algorithm. This approach requires a sample of 100 annotated sentences for head selection and raises F1 The extraction of constituency trees from language models was described by Kim et al. [13]. They present a comprehensive study that covers nine types of pretrained networks: BERT (base, large), GPT-2 [30] (original, medium), RoBERTa [19] (base, large), XLNet [38] (base, large). Their approach is based on computing distance between each pair of subsequent words. In each step, they are branching the tree in the place where the distance is the highest. The authors try three distance measures on the vector outputs of the encoder layer (cosine, L1, and L2 distances for pairs of vectors) and two distance measures on the distributions of token's attention (Jason-Shannon and Hellinger distances for pairs of distribution). In the former case, distances are computed only per layer and in the latter case for each head and average of heads in one layer. The best setting achieves 40.1% F1 score on WSJ Penn Treebank. It uses XLNet-base and Helinger distance on averaged attentions in the 7th layer. Generally, attention distribution distances perform better than vector ones. Authors also observe that models trained on regular language modeling objective (i.e., next word prediction in GPT, XL-Net) captured syntax better than masked language models (BERT, RoBERTa). In line with the previous research, the middle layers tend to be more syntactic.
Figure 8 summarizes the evaluation of syntactic information across layers for different approaches. In Transformerbased language models: BERT, mBERT, and GPT-2, the middle layers are the most syntactic. In neural machine translation models, the top layers of the encoder are the most syntactic. However, it is important to note that the [12,6]. The method D) shows the dependency alignment averaged across all heads in each layer [34]. The methods E) and F) show UAS of trees induced from attention heads by the maximum spanning tree algorithm [31,15]. The results for the best layer (corresponding to value 1.0 in the plot) are: A) 82.5; B) 79.8; C) 80.1; D) 22.3; E) 24.3; F) en2cs: 23.9, en2de: 20.9, en2et: 22.1, en2fi: 24.0, en2ru: 22.4, en2tr: 17.5, en2zh: 21.6; G) 77.0 NMT Transformer encoder is only the first half of the whole translation architecture, and therefore the most syntactic layers are, in fact, in the middle of the process. In RNN language model (ELMo) the first layer is more syntactic than the second one.
We conjecture that the initial Transformer's layers capture simple relations (e.g., attending to next or previous tokens) and the last layers mostly capture task-specific information. Therefore, they are less syntactic.
We also observe that in supervised probing [12,6], better results are obtained from initial and top layers than in unsupervised structure induction [31,15], i.e., the distribution across layers is smoother.
In this overview, we survey that syntactic structures are latently learned by the neural models for natural language processing tasks. We have compared multiple approaches of others and described the features that affect the ability to capture the syntax. The following aspects tend to improve the performance on syntactic tasks such as POS tagging:
1. Using contextual embeddings from RNNs or Transformer outperforms static word embeddings (Word2Vec, GloVe).
2. Pretraining on tasks with masked input (language modeling or machine translation) produces better syntactic representation than auto encoding. 3. The advantage of language modeling over machine translation is the fact that larger corpora are available for pretraining. Our meta-analysis of latent states showed that the most syntactic representation could be found in the middle layers of the model. They tend to capture more complex relations than initial layers, and the representations are less dependent on the pretraining objectives than in the top layers.
We have shown to what extent systems trained for a nonsyntactic task can learn grammatical structures. The question we leave for further research is whether providing explicit syntactic information to the model can improve its performance on other NLP tasks.
This work has been supported by the grant 18-02196S of the Czech Science Foundation. It has been using language resources and tools developed, stored and distributed by theLINDAT/CLARIAH-CZ project of the Ministry of Education, Youth and Sports of the Czech Republic (project LM2018101).