Attention-based deep learning models have demonstrated significant improvement over traditional algorithms in several NLP tasks. The Transformer, for instance, is an illustrative example that generates abstract representations of tokens that are input to an encoder based on their relationships to all tokens in a sequence. While recent studies have shown that such models are capable of learning syntactic features purely by seeing examples, we hypothesize that explicitly feeding this information to deep learning models can significantly enhance their performance in many cases. Leveraging syntactic information like part of speech (POS) may be particularly beneficial in limited-training-data settings for complex models such as the Transformer. In this paper, we verify this hypothesis by infusing syntactic knowledge into the Transformer. We find that this syntax-infused Transformer achieves an improvement of 0.7 BLEU when trained on the full WMT '14 English to German translation dataset and a maximum improvement of 1.99 BLEU points when trained on a fraction of the dataset. In addition, we find that the incorporation of syntax into BERT fine-tuning outperforms BERT BASE on all downstream tasks from the GLUE benchmark, including an improvement of 0.8% on CoLA.
like syntax and semantics, but this knowledge is imperfect [3]. However, works such as [4], [5] show that NLP Attention-based deep learning models for natural lan- models that acquire even a weak understanding of syntacguage processing (NLP) have shown promise for various tic structure through training demonstrate improved permachine translation and natural language understanding formance relative to their baseline counterparts. Hence, tasks. For word-level sequence-to-sequence tasks such we hypothesize that explicit prior knowledge of syntactic as translation, paraphrasing, and text summarization, information can benefit NLP models in various tasks. attention-based models allow a single token (in our case, The Transformer relies on a significant amount of data subwords) to be represented as a combination of all to- and extensive training to accurately pick up on syntactic kens in the sequence [1]. The distributed context allows and semantic relationships [3]. Previous studies have attention-based models to infer rich representations for shown that an NLP model’s performance improves with tokens, leading to more robust performance. One such the ability to learn the underlying grammatical structure model is the Transformer, which features a multi-headed of a sentence [4, 5, 6]. It has been shown that simultaself- and cross-attention mechanism that allows many neously training models for machine translation, part of diferent representations to be learned for a given token speech (POS) tagging, and named entity recognition proin parallel [2]. The encoder and decoder arms contain vide a slight improvement over baseline on each task for several identical stacked subunits to learn embeddings small datasets [7]. Inspired by these previous eforts, we for tokens in the source and target vocabularies. propose to utilize external knowledge, namely the syn
Syntax is an essential feature of grammar from ex- tactic features inherent in natural language sequences, ternal knowledge that facilitates generation of coherent to enhance the performance of the Transformer model. sentences. For instance, POS dictates how words relate to We suggest a modification to the embeddings fed into one another (.., verbs represent the actions of nouns, the Transformer architecture that allows the tokens input adjectives describe nouns, etc.). Studies show that when to the encoder to attend not only to other tokens but also trained for a suficiently large number of steps, NLP mod- syntactic features including POS, case (upper or lower), els can potentially learn underlying patterns about text and subword position [8]. Like POS, case is a categorical feature that can allow the model to distinguish common words from important ones. Subword tags can help bring cohesion among subwords of a complex word (say ‘amal’, ‘ga’, ‘mation’ of “amalgamation”) so that their identity $ ds448@duke.edu (D. Sundararaman) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org) as a unit is not compromised by tokenization. These fea- a form of relative positional embedding by enforcing the tures are identified using a separate model (for POS) or sentence structure. [11] encourages diferent attention are directly specified (for case and subword position) and heads to learn about diferent information like position are appended to the one-hot vector encoding for each to- and output representation by introducing a disagreement ken. For example, the feature-augmented subword tokens regularization. To model the local dependency between of ‘amalgamation’ would look like: ‘amal,NOUN,0,S’, words more eficiently, [ 12] introduced distance between ‘ga,NOUN,0,M’, ‘mation,NOUN,0,E’. Here, ‘NOUN’ in- words and incorporated it into the self-attention. dicates the POS; 0 indicates lowercase; and ‘S’, ‘M’, and Previous literature also has sought to incorporate syn‘E’ indicates that the subwords occur at the start, middle, tax into deep learning NLP models. [13] used syntax deand end of the word, respectively. Embeddings for the pendency tree on a bidirectional RNN on translation systokens and their features are learned jointly during the tems by modeling the trees using Graph Convolutional training. As the embeddings pass the Transformer layers, Networks (GCNs) [14]. Incorporating syntax informathe representation for each token is synthesized using a tion by linearizing parse trees and adding a syntax-based combination of subword token and syntactic features. distance constraint on the attention module helped sig
We apply our approach to English to German (EN-DE) nificantly in Chinese-English and English-German transtranslation on the WMT ’14 dataset and demonstrate that lation [15]. the BLEU score of the feature-rich syntax-infused Trans- Finally, [16] has incorporated source side syntax on former uniformly outperforms the baseline Transformer Transformer encoders in a multi-task setup with parse as a function of training data size. We also evaluate the trees. However, in this approach, the Transformer strugproposed model on two low-resource languages – Korean gles to learn to parse and decode the syntactic sequences and Sinhala – where the source translation language is and only showed performance gains in low-resource always English. Finally, we experiment with this modi- languages. We append the syntactic information as seifcation of embeddings on the BERT BASE model on sev- quences and decode only the target sequence, thus not eral General Language Understanding Evaluation (GLUE) complicating the Transformer’s job. These works afirm benchmarks and observe considerable improvement in that adding syntax information can help the NLP modperformance on all the tasks. Examining the attention els translate better from one language to another and weights learned by the proposed model indicates that the achieve better performance measures. attention is more dispersed than the baseline model.
To summarize, our contributions are as follows: 3. Background
dings of the Transformer model, incorporating external syntax information, including POS, case, and subword position. 2. We demonstrate superior performance on the
WMT ’14 EN-DE machine translation task and for low-resource machine translation. 3. We infuse pre-trained BERTBASE embeddings with syntax information and find that our model outperforms BERTBASE on all of the GLUE benchmark tasks.
The Transformer consists of encoder and decoder modules, each containing several subunits that act sequentially to generate abstract representations for words in the source and target sequences [2]. For all ∈ {1, 2, . . . , }, where is the length of the source sequence, the encoder embedding layer first converts tokens x into embeddings e: e = Ex where E ∈ R× is a trainable matrix with column constituting the embedding for token , is the total number of tokens in the shared vocabulary, and x ∈ {0, 1} : 2. Related Work ∑︀ = 1 is a one-hot vector corresponding to token . These embeddings are passed sequentially through Previous works have sought to improve the self-attention six encoder subunits. Each of these subunits features a module to aid NLP models. For instance, [9] introduced self-attention mechanism, that allows tokens in the input a Gaussian bias to model locality, to enhance model abil- sequence to be represented as a combination of all tokens ity to capture local context while also maintaining the in the sequence. Attention is accomplished using three long-range dependency. Instead of absolute positional sets of weights: the key, query, and value matrices (K, embeddings, [10] experimented with relative positional Q, and V, respectively). The key and query matrices embeddings or distance between sequences and found interact to score each token in relation to other tokens, that it led to a significant improvement in performance. and the value matrix gives the weights to which the score
Adding linguistic structure to models like the Trans- is applied to generate output embedding of a given token. former can be thought of as a way of improving the attention mechanism. The POS and subword tags act as
K = HW Q = HW V = HW A = softmax ︂( QK⊤ )︂ √
V (1) where H = [h1 h2 · · · h ]⊤ ∈ R× are the dimensional embeddings for a sequence of tokens indexed by ; W , W, and W all ∈ R× are the projection matrices for keys, queries, and values, respectively; is a scaling constant (here, taken to be ) and A ∈ R× is the attention-weighted representation of each token. Note that these are subunit-specific – a separate attention-weighted representation is generated by each subunit and passed on to the next. Moreover, for the first layer, h := e.
The final subunit then passes its information to the decoder, which also consists of six identical subunits that behave similarly to those of the encoder. One key diference between the encoder and decoder is that the decoder not only features self-attention but also crossattention; thus, when generating new words, the decoder pays attention to the entire input sequence as well as to previously decoded words. 3.2. BERT coder can also be fine-tuned to perform a number of diferent downstream tasks. Bidirectional encoder representations of Transformers (BERT) [17] is an extension of the Transformer model that allows for such fine-tuning. The BERT model is essentially a Transformer encoder (with number of layers , embedding dimension , and number of attention heads ) which is pre-trained using two methods: masked language modeling (MLM) and next-sentence prediction (NSP). Subsequently, a softmax layer is added, allowing the model to perform various tasks such as classification, sequence labeling, question answering, and language inference. 4. Knowledge-Infused Models
tactic information for better translation, we () employ a pre-trained model1 with high accuracy to tag words in the source sequence with their POS, () identify the case of each word, and () identify the position of each
The token embeddings learned by the Transformer en- Figure 1: (a) Formation of attention matrices (K, Q, and V) To aid the Transformer in acquiring and utilizing syn- sequence is first associated with its POS label according (a) Syntax-infused Transformer
(b) Syntax-infused BERT with syntactic information. The left column shows the token embedding matrix; the embedding matrices for the various features are shown on top. Embeddings for the features are either concatenated or summed (denoted by ⊕ ) and finally, concatenated to the token embeddings. Multiplication with learned weights results in K, Q, and V. The attention matrices are double shaded to indicate the mix of token and syntax information. (b)The BERTBASE + POS model. Token embeddings are combined with trainable POS embeddings and fed into the BERT encoder. The final embedding of the [CLS] token is fed into a softmax for downstream classification tasks. subword relative to other subwords that are part of the same word (subword tagging). We then append trainable syntax embedding vectors to the token embeddings, resulting in a combined representation of syntactic and semantic elements. Specifically, each word in the source to syntactic structure. We then assign each subword token the POS label of the word from which it originated.
words sun, sh, and ine, each subword token would be assigned the POS NOUN. The POS embeddings f ∈ R for each token (indexed by ) are then extracted from a trainable embedding matrix using a look-up table. 4.2. Syntax-infused BERT
In a similar manner, we extract case and subword position features. For case, we use a binary element Adding syntactic features to the BERT model is a natural for e∈ac{h0,su1}bwtoorldo,okdeuppenadfienagtuorne wemhbetehdedrinthgefori∈ginRal eFxotreansgiiovnenoftotkheena,bitosvienpmuotdrieficparteiosennttoattihoen Tisracnonsfsotrrumcetre.d word is capitalized. For subword position, we use a cate- by summing the corresponding BERT token embeddings with POS embeddings (see Figure 1b). Mathematically, gorical element ∈ {, , , } to identify a feature embedding f ∈ R for each subword depending on the input tokens h′ ∈ R are given by h′ = e + f, whether the subword is at the start (), middle ( ), or where e is the BERT token embedding and f is the end () of the word; if the subword comprises the full POS embedding for token . For single sequence tasks, word, it is given a tag of . These are then added onto = 1, 2, . . . , , where is the number of tokens the POS embedding and concatenated with the token em- in the sequence; while for paired sequence tasks, = beddings e ∈ R− to create a combined embedding 1, 2, . . . , 1 + 2, where 1 and 2 are the number (see Figure 1a). Mathematically, in the input stage, h of tokens in each sequence. As is standard with BERT, becomes: for downstream classification tasks, the final embedded [e⊤ f⊤]⊤ = h′ ∈ R irseppraessseendtathtiroonuyg^hasoofft mthaexfircstlatsoskifieernt(odegneonteerdataesa[ClaLbSe]l).
We modify the BERTBASE model using this approach and denote it as BERT BASE + POS. where f = f + f + f ∈ R is the learned embedding for the syntactic features of subword in the sequence of subwords. The augmented embeddings are then passed through the Transformer’s multi-headed attention, layer normalization, and feedforward modules, resulting in a syntax-infused hidden state of each encoder layer: (|1:− 1, z′,) = softmax(Scores()) 5.1. Machine translation where is the decoder, which undergoes the same pro- For EN-DE, we train both the baseline and syntax-infused cess outlined in equation 2, and comprises cross-attention Transformer for 100,000 steps. All hyperparameter setwith the encoder output and a causal self-attention (pa- tings of the baseline Transformer, including embedding rameters of the decoder omitted for conciseness). = dimensions of the encoder and decoder, match those of 1, . . . , indexes decoded tokens , and W is a linear [2]. We train the syntax-infused Transformer model usprojection matrix mapping the decoder output to the vo- ing 512-dimensional embedding vectors. In the encoder, cabulary size. The scores for output tokens are now a = 492 dimensions are allocated for subword token emfunction of z′,, which is aware of the source syntax.
A˜ = [A0 · · · H˜ = [H′ · · ·
A7]W
H′] Z˜ = LayerNorm(A˜ + H˜)
(2)
Z = LayerNorm(Z˜ + FeedForward(Z˜)) where H˜ are the syntax-augmented embeddings, repeated eight times (once for each attention head); A˜ is the concatenation of the attention-weighted representation of the output of the eight heads; and Z is the output of layer . LayerNorm and FeedForward denote the layer normalization function and feedforward neural network layers, respectively, and are applied separately for each head. This is repeated times in layers, with the output of the previous layer treated as the input to the next. The final syntax-infused encoder output Z is attended to by the decoder, aiding the autoregressive text generation process in the final layer of the Transformer:
z′, = (Z, 1:− 1) Scores() = LayerNorm(z′,)W (3)
DE (German) dataset, which consists of 4.5M training sentence pairs. Validation is performed on newstest2013 (3000 sentence pairs), and testing is on the newstest2014 dataset (2737 sentence pairs, [18]). For low-resource machine translation (1M sentence pairs or less), we employ the OpenSubtitles EN-KO (Korean) and EN-SI (Sinhala) datasets [19]. The latter was found by [20] to be a lowresource language with morphology and syntax significantly diferent from that of English. We subsample these datasets from 1.3M and 601K sentence pairs, respectively, to 500K sentence pairs, and train on 90%, validate on 9%, and test on 1%. One of the key reasons to incorporate POS features to EN sequences is that parsers that infer syntax from EN sentences are typically trained on a greater number and variety of sentences and are therefore more robust than parsers for other languages. Unlike [16], in our experiments, we always translate sequences from EN to other languages as the objective is to improve the translation performance and not to decode the POS sequences.
Data Fraction 1% 5% 10% 25% 50% 100%
Number of Sentences 45k 225k 450k 1.1M 2.25M 4.5M
Baseline Transformer 1.10 8.51 16.28 22.72 25.41 28.94
Syntax-infused Transformer 1.67 10.50 17.28 23.24 25.74 29.64 decoder layers, with eight heads for multi-headed attention. Parameters are initialized from a Glorot uniform distribution [21]. We use a dropout rate of 0.1 and batch size of 4096. We utilize the Adam optimizer to train the model with 1 = 0.9 and 2 = 0.998; gradients are accumulated for two batches before updating parameters. A label-smoothing factor of 0.1 is employed.
The training settings are exactly the same for lowresource translation except that we train the Transformer models for 50,000 steps owing to the reduced data complexity. The dimension of features f is chosen to be 20 by doing a grid search over the range of 8 to 64.
(a) Baseline (EN-DE) (b) Syntax-infused (EN-DE)
BLEU scores improvements on low resource languages includ- Multi-Genre Natural Language Inference (MNLI), Quora ing Sinhala and Korean for the baseline and syntax-infused Question Pairs (QQP), Question Natural Language Infermodels. ence (QNLI), Stanford Sentiment Treebank (SST-2), the Corpus of Linguistic Acceptability (CoLA), the Semantic Textual Similarity Benchmark (STS-B), Microsoft Rebeddings while = 20 for feature embeddings (chosen search Paraphrase Corpus (MRPC), and Recognizing Texby hyperparameter tuning). In the decoder, all 512 di- tual Entailment (RTE). For a summary of these datasets, mensions are used for subword token embeddings (since see [17]. We use POS as the syntactic feature for BERT for we are decoding words, not word-POS pairs). these tasks. Aside from the learning rate, we use identical The model architecture consists of six encoder and six hyperparameter settings to fine-tune both the BERT BASE Reference Parken in Frankfurt könnte bald empfindlich teurer werden .
Die zurückgerufenen Modelle wurden zwischen dem 1. August und 10. September hergestellt .
Stattdessen verbrachte Bwelle Jahre damit , seinen Vater in überfüllte Kliniken und Hospitäler zu begleiten , um dort die Behandlung zu bekommen , die sie zu bieten hatten .
Patek kann gegen sein Urteil noch Berufung ein legen .
Baseline Transformer Das Personal war sehr freundlich und hilfsbereit .
Zwischen August 1 und September 10.
Stattdessen verbrachte Bwelle Jahre damit , seinen Vater mit über füllten Kliniken und Krankenhq¨usern zu beherbergen .
Patek kann noch seinen Satz an rufen .
Syntax-infused Transformer Parken in Frankfurt könnte bald spürbar teurer sein .
Die zurückgerufenen Modelle wurden zwischen dem 1. August und 10. September gebaut Stattdessen verbrachte Bwelle Jahre damit , seinen Vater zu überfüllten Kliniken und Krankenhäusern zu begleiten , um jede Behandlung zu bekommen , die sie bekommen konnten .
Patek mag sein Urteil noch Berufung ein legen . and BERTBASE + POS models for each task. This includes a through grid search). This means that (1) each feature batch size of 32 and 3 epochs of training for all tasks. For embedding dimension can be allocated to 20 and summed each model, we also choose a task-specific learning rate together or (2) the feature embeddings can be concateamong the values {5, 4, 3, 2} × 10− 5, which is standard nated to each other such that their total dimension is 20. for BERTBASE. Therefore, in order to eficiently learn the feature embeddings while also not sacrificing the token embedding dimensionality, we find that summing the embeddings for 6. Experimental Results all three diferent features of = 20 and concatenating the sum to the subword token embeddings of = 492 6.1. Machine translation gives the maximum performance on translation. We evaluate the impact of infusing syntax into the base- We report results on the WMT ’14 EN-DE translation line Transformer on the WMT ’14 EN-DE translation task task in Table 1. We vary the proportion of data used as well as on the low resource EN-SI and EN-KO language for training and observe the performance of both the pairs. There are multiple ways to incorporate feature em- baseline and syntax-infused Transformer. The syntaxbeddings into the subword token embeddings, such as infused model markedly outperforms the baseline model, direct summation, vector projection, or concatenation. ofering an improvement of 0.57, 1.99, 1, 0.52, 0.33, and For a fair comparison to the baseline Transformer, we 0.7 points, respectively, when trained on 1, 5, 10, 25, 50, ifx a total of 512 dimensions for representing both the and 100% of the full training data. It is notable that the token embeddings and feature embeddings. One impor- syntax-infused model translates the best relative to the tant tradeof is that as the dimensionality of the syntax baseline when only a fraction of the dataset is used for information increases, the dimensionality for token em- training. Specifically, the maximum improvement is 1.99 beddings decreases. Since POS, case, and subword tags BLEU points when only 10% of the training data is used. have only a limited number of values they can take, ded- This shows that explicit syntax information is most helpicating a high dimensionality for each feature proves ful under limited training data conditions. As shown in detrimental. We find that the total feature dimension for Figure 2(a)-(b), the syntax-infused model is better able which the gain in BLEU score is maximized is 20 (found to capture connections between tokens that are far apart Sentence 1 The Qin (from which the name China is derived) established the approximate boundaries and basic administrative system that all subsequent dynasties were to follow .
Steve Jobs was attacked by Sculley and other Apple executives for not delivering enough hot new products and resigned from the company a few weeks later.
Sentence 2 Qin Shi Huang was the first Chinese Emperor .
Steve Jobs worked for Apple.
True label Not entailment Entailment yet semantically related, resulting in improved transla- the RTE dataset, see Table 5. tion performance. In addition, Table 3 shows a set of sample German predictions made by the baseline and syntax-infused Transformer. 7. Conclusions
In Table 2, we compare the performance of baseline and syntax-infused Transformers on low resource lan- Infusing external syntax into NLP models has proven to guages, namely Korean and Sinhala. These languages be an efective method to improve performance on generare often under represented in the NLP literature and ation as well as classification tasks. In this work, we have fall under categories of languages with limited training infused the Transformer and BERT models with syntactic data. Results confirm the efectiveness of our method- features including POS, word case, and subword position ology of adding syntax to source sequences to improve using a simple concatenation of token embeddings and eficiency, irrespective of the training data size. In ad- trainable feature embeddings from external knowledge. dition, while we find there are improvements in BLEU We conducted experiments on the large-scale WMT ’14 scores with the syntax-infused Transformer, Table 2 also EN-DE translation task and low-resource EN-KO and shows the efect of addition of syntax to RNN-based net- EN-SI OpenSubtitles datasets. We found significant imworks for the same translation tasks leads to a drop in provements in EN-DE translation, especially as the size performance. For this experiment, we employed an 2- of the training dataset is reduced. These results were layer LSTM encoder and 2-layer LSTM decoder with 500 confirmed in the low-resource setting, suggesting that hidden states each and augmented encoder tokens with our method performs well regardless of the training data syntax features in the same way as for the Transformer. size, a feature that makes it less rely on the training [5] confirms our observation that recurrent structures data. We then modified BERT by adding trainable synstruggle to learn the subject-verb agreement and that tactic embeddings directly to the input and found that better architectures may be required to retain the com- our BERTBASE + POS model performs better than baseline plex syntactic structures present in the languages. On on a number of GLUE downstream tasks. Thus, we have the other hand, Transformers have been known to learn proven both the text generation and classification applisyntax with suficient training and an increased amount cations benefits from knowledge injection. of data. Studies [23] have also shown that explicit syntax While many existing works either suggest that Transsupervision in Transformers yield superior results. formers are capable of learning syntax implicitly or attempt to train Transformers to predict syntax in a multitask setting, our results suggest that, when available, it 6.2. Natural language understanding is preferable to directly incorporate syntactic features as Results obtained for the BERTBASE + POS model on the inputs to the NLP model to improve performance. As GLUE benchmark test set are presented in Table 4. a next step, we plan to study the alignment of tokens BERTBASE + POS outperforms BERTBASE on all tasks, with between source and target sequences to determine the the exception of matched MNLI. The improvements range efect of adding syntax into the model. In addition, we from marginal to significant, with a maximum improve- plan to investigate syntax infusion into other NLP tasks ment of 0.8 points of the POS model over BERTBASE on such as co-reference resolution. In this context, syntactic CoLA. Fittingly, CoLA is a task which assesses the linguis- information could be helpful in matching tokens to their tic structure of a sentence, which is explicitly informed by antecedents by virtue of parts of speech, for instance. POS embeddings. Moreover, BERTBASE + POS outperforms BERTBASE on tasks that are concerned with evaluating semantic relatedness. For examples of predictions where BERTBASE + POS performs better than BERTBASE made on [15] J. Li, D. Xiong, Z. Tu, M. Zhu, M. Zhang, G. Zhou,
Modeling source syntax for neural machine trans[1] M.-T. Luong, H. Pham, C. D. Manning, Efective ap- lation, arXiv preprint arXiv:1705.01020 (2017). proaches to attention-based neural machine trans- [16] A. Currey, K. Heafield, Incorporating source synlation, arXiv preprint arXiv:1508.04025 (2015). tax into transformer-based neural machine transla[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, tion, in: Proceedings of the Fourth Conference on L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, At- Machine Translation (Volume 1: Research Papers), tention is all you need, in: Advances in neural 2019, pp. 24–33. information processing systems, 2017, pp. 5998– [17] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, 6008. Bert: Pre-training of deep bidirectional transform[3] G. Jawahar, B. Sagot, D. Seddah, S. Unicomb, ers for language understanding, arXiv preprint G. Iñiguez, M. Karsai, Y. Léo, M. Karsai, C. Sarraute, arXiv:1810.04805 (2018). É. Fleury, et al., What does bert learn about the [18] B. Zhang, I. Titov, R. Sennrich, Improving deep structure of language?, in: 57th Annual Meeting transformer with depth-scaled initialization and of the Association for Computational Linguistics merged attention, arXiv preprint arXiv:1908.11365 (ACL), Florence, Italy, 2019. (2019). [4] A. Kuncoro, C. Dyer, J. Hale, D. Yogatama, S. Clark, [19] P. Lison, J. Tiedemann, Opensubtitles2016: ExtractP. Blunsom, Lstms can learn syntax-sensitive depen- ing large parallel corpora from movie and tv subtidencies well, but modeling structure makes them tles (2016). better, in: Proceedings of the 56th Annual Meeting [20] F. Guzmán, P.-J. Chen, M. Ott, J. Pino, G. Lample, of the Association for Computational Linguistics P. Koehn, V. Chaudhary, M. Ranzato, The flores (Volume 1: Long Papers), 2018, pp. 1426–1436. evaluation datasets for low-resource machine trans[5] T. Linzen, E. Dupoux, Y. Goldberg, Assessing the lation: Nepali-english and sinhala-english, arXiv ability of lstms to learn syntax-sensitive dependen- preprint arXiv:1902.01382 (2019). cies, Transactions of the Association for Computa- [21] X. Glorot, Y. Bengio, Understanding the dificulty tional Linguistics 4 (2016) 521–535. of training deep feedforward neural networks, in: [6] H. Chen, S. Huang, D. Chiang, J. Chen, Improved Proceedings of the thirteenth international conferneural machine translation with a syntax-aware en- ence on artificial intelligence and statistics, 2010, coder and decoder, arXiv preprint arXiv:1707.05436 pp. 249–256.
(2017). [22] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S. R. [7] J. Niehues, E. Cho, Exploiting linguistic resources Bowman, Glue: A multi-task benchmark and analfor neural machine translation using multi-task ysis platform for natural language understanding, learning, arXiv preprint arXiv:1708.00993 (2017). arXiv preprint arXiv:1804.07461 (2018). [8] R. Sennrich, B. Haddow, Linguistic input features [23] C. Wang, S. Wu, S. Liu, Source dependency-aware improve neural machine translation, arXiv preprint transformer with supervised self-attention, arXiv arXiv:1606.02892 (2016). preprint arXiv:1909.02273 (2019). [9] B. Yang, Z. Tu, D. F. Wong, F. Meng, L. S. Chao,
T. Zhang, Modeling localness for self-attention networks, arXiv preprint arXiv:1810.10182 (2018). [10] P. Shaw, J. Uszkoreit, A. Vaswani, Self-attention with relative position representations, arXiv preprint arXiv:1803.02155 (2018). [11] J. Li, Z. Tu, B. Yang, M. R. Lyu, T. Zhang, Multi-head attention with disagreement regularization, arXiv preprint arXiv:1810.10183 (2018). [12] J. Im, S. Cho, Distance-based self-attention network for natural language inference, arXiv preprint arXiv:1712.02047 (2017). [13] J. Bastings, I. Titov, W. Aziz, D. Marcheggiani,
K. Sima’an, Graph convolutional encoders for syntax-aware neural machine translation, arXiv preprint arXiv:1704.04675 (2017). [14] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, arXiv preprint arXiv:1609.02907 (2016).