These improvements open many possibilities in solving Natural Language Processing downstream tasks. Such tasks include machine translation, speech recognition, information retrieval, sentiment analysis, summarization, question answering, multilingual dialogue systems development, and many more. Language models are one of the most important components in solving each of the mentioned tasks. This paper is devoted to research and analysis of the most adopted techniques and designs for building and training language models that show a state of the art results. Techniques and components applied in the creation of language models and its parts are observed in this paper, paying attention to neural networks, embedding mechanisms, bidirectionality, encoder and decoder architecture, attention, and self-attention, as well as parallelization through using transformer. As a result, the most promising techniques imply pre-training and fine-tuning of a language model, attention-based neural network as a part of model design, and a complex ensemble of multidimensional embedding to build deep context understanding. The latest offered architectures based on these approaches require a lot of computational power for training language models, and it is a direction of further improvement. Algorithm for choosing right model for relevant business task provided considering current challenges and available architectures.
Natural Language Processing (NLP) is computer comprehension, analysis, manipulation, and generation of natural language. NLP covers many different applications like machine translation, speech recognition, optical character recognition, part of speech tagging, information retrieval, summarization, question answering, dialog systems building, and many more. Since the last decade, there have been a great number of breakthroughs towards making machines understand the language (text or voice) better that raised enormous interest in this field of scientific research. The highest goal for all scientists working in the area of NLP is to build such techniques that will allow computers to comprehend natural language as text or voice at a human level which is not reached yet. Once achieved, computational systems will be able to understand and generate accurate human-like language, be it a text or an audible language. This paper analyses widespread techniques and components in building language models to give a scientist thorough information for further research and improvement. There have been a lot of discoveries of language models from computational linguistics scientists, and those new techniques showed great results on specific tasks. But when it comes to the broad spectrum of NLP tasks solved by the same language model not many show the same high results. The recent state of the art architectures (BERT, RoBERTa, Transformer-XL, XLNet, etc.) leverage the following approaches: contextual embedding, bidirectionality, encoder-decoder architecture, attention mechanism and transformer, pretraining modeling and fine-tuning.
The structure of this paper includes language models architecture observation in Sect. 2. Word and contextual embedding techniques are described in Sect. 3 and encoder-decoder—in Sect. 4. Neural Networks applied as a part of encoder and decoder are observed in Sect. 5. Model choosing algorithm presented in Sect.6. Conclusion in Sect. 7 provides further promising areas of research in NLP. 2
NLP downstream tasks (machine translation, sentiment analysis, question answering,
part of speech tagging, and many more) usually are solved with some different
approaches chosen for a specific task. Generally, it comes to supervised learning on
taskspecific datasets, which is quite consuming in terms of research hours and
computational resources. In addition to these inconveniences, systems that are built with this
approach are very sensitive to task specifications and changes in the data distribution.
Current trends move towards unsupervised universal models and transfer learning as
pre-trained models with further fine-tuning. The techniques and components that are
offered for observation in this article correspond to current trends and groundbreaking
achievements in NLP [
For a neural network to be able to complete its task there is a necessity to provide a
numerical token representation of input sequence. Word embedding techniques create
vectors out of tokens. Vectors comparison results in tokens semantic similarity.
Embedding techniques such as GloVe and Word to vector explain the concept of modeling
input sequence through representation [
The main idea and task are to represent and map words (documents, phrases, context,
a piece of a word, or a character) as a vector of numbers to use probability distributions
or likelihoods of tokens in language corpora to separate semantic similarity categories.
Hence different words with similar meanings will have similar vectors and different by
meaning groups of words should be separable in vector space. The underlying idea that
“a word is characterized by the company it keeps” was popularized by Firth [
The neural network encoder-decoder model significantly improved the performance of language models. Quite simple Recurrent Neural Network (RNN) architecture to process input and output sequences of variable lengths was offered. The input sequence of words [“How,” “are,” “you,” “?”] go through the embedding layer to get numerical representation, after which numerical representation goes sequentially to RNN. RNN process input embedding sequentially (from left to right) passing to the next timestamp RNN hidden state calculated in the current timestamp RNN. After all inputs proceed to the final time stamp, the final timestamp RNN produces output representing all input sequences in one hidden state. This part called encoder as the main task is not to generate predictions but to encode input sequences. After encoder finishes to encode, hidden state passes to the decoder which task is to decode and generate predictions based on input hidden state. Decoder process sequentially taking as input to each current timestamp output activation of previous timestamp RNN and output prediction of previous timestamp RNN. For the first timestamp, it takes a beginning of sentence token (BOS) as a prediction of the previous layer. The decoder generates predictions until it generates end of sentence token (EOS, depending on implementation can be until some length or different parameter).
The strongest part of this approach is the ability to train an end-to-end model right on the source and target data as well as the possibility to handle input and output sequences of different lengths. Therefore, that it resolves the problem of different lengths of an input and an output sequence in Neural Machine Translation.
The encoder-decoder architecture consists of two RNNs or more often Long ShortTerm Memory (or Gated Recurrent Unit) to avoid the problem of vanishing gradient covered later in this article. The encoder encodes all input sequences and stores all information in context or encoder vector (in simplest architecture last hidden state used) that is input to decoder, which decodes by result predictions. 5
Progress in the application of Neural Networks to NLP tasks brings huge improvements in both science and business areas. 5.1
RNNs [
The main idea behind RNNs [
yt g Wyht by .
Especially important for NLP areas is that the output vector’s contents are calculated not only by the one current input but based on the entire history of inputs that network processed in the past. RNN cells take the output of the previous cell as an input to the current cell, and the previous cell contains information of its previous cell and so on. This type of connection is called a recurrent connection.
Despite the wide adoption, RNN possesses significant drawbacks [
In response to medium and long-range dependency problems researchers propose two architectures, with the core idea of Cell State (kind of residual connection).
The main idea in Gated Recurrent Unit (GRU) [
These Update and Reset Gates are the key concepts behind GRU and dealing with dependencies problems of basic RNNs. ht-1
GRU cell xt rt
Reset gate zt 1 Update gate tanh h~t ht
Another type of architecture that can capture mid-term dependencies, even more
powerfully than GRU, is Long Short-Term Memory (LSTM) [
LSTM, also, uses two separate gates (Update Gate and a Forget Gate) to update Memory Cell value, instead of using single Update Gate in GRU (that either keep or forget previous memory cell value). And instead of using Reset Gate in Candidate Value, it uses element-wise multiplied with Memory Cell value as shown in Fig. 2. Input Gate it (xtU i ht 1W i ), decides which information crucial to keep and Forget Gate ft (xtU f ht1W f ), decides which and how much information not to keep, in other words, to forget (which might be intersecting or not with input gate). Input and Forget gates do this using previous time step hidden state ht-1 and current input xt. Both gates using sigmoid activation function, which gives possibility in most cases to have values of gates either close to 0 or 1.
Usage of separate Update Gate and Forget Gate to calculate Memory Cell value = ( ∗ −1 + ∗ ̃ ) gives Memory Cell the possibility not only to store new information in the current time step Memory Cell by using Candidate Memory Cell ( ̃ ), but also the option to keep some amount of information from previous time step Memory Cell (Ct-1). Output Gate ot (xtU 0 ht 1W 0 ), at the end uses to calculate the current time step output hidden state ht tanh(Ct ) ot , based on updated Memory Cell value calculated before.
The state of a cell is straight forward. It flows down the whole unit with minor linear changes. This is why two proposed architectures were very good at memorizing longterm dependencies.
Ct-1 ht-1
LSTM cell ft it ~ Ct tanh tanh
ht Xt
Foget gate
Input gate
Output gate
Ct ht
These networks are quite consuming for computational resources. Moreover, this type of architecture cannot be parallelized: hence, it is very expensive to train on a big corpus of data. And despite the fact it works much better with longer sequences there is a noticeable loss in sequences with more than 20 words. Retrospectively application of RNN, GRU, and LSTM was the major stage in modern NLP that significantly affected future development of the area. Additionally, there was a significant amount of work on the bidirectionality of RNN to provide models with the possibility to capture and use information from both earlier and later in the sequence.
If to express in simple words Bidirectional RNN (BRNN) [
To construct even more powerful models researchers propose to stack units of RNN/LSTM/GRU. This type of architecture is called Deep RNN. The bottleneck of BRNN is that it needs the entire sequence of data before making any predictions. Deep RNN is also much more expensive in computation. All of the networks presented had problems in neural machine translation as the input and output sequences regularly were of different lengths because of different language semantics. 5.4
The focus of researchers was the problem of long sentences (sentence contained more
than 20 words) which cannot be stored effectively in one output vector of
RNN/GRU/LSTM. In [
The main intuition behind Attention is that humans do not read and memorize whole long sentences at once, but part by part. And for a decoder, it would be valuable to know while decoding (for example translation), to which part of the input sequence it should pay more attention. An idea of attention: at each step, the decoder focuses on some particular part of the source. Decoder focuses only on particular words at each step (increased saturation represents more attention), not on the full input sequence.
Attention mechanism uncovers such possibility to a decoder by Attention Weights and Context Vector.
In addition to BRNN (which can also be BGRU, BLSTM) Attention concept utilizes the idea of alignment scores and attention weights (the amount of attention decoder should pay while calculating current time step prediction).
The all-time step hidden states of encoder pass with the last layer hidden state of encoder to the decoder. Central processing occurs in the decoder. Each time step of decoder, a set of features (about words and surrounding words) computes and called Alignment Scores eij a(si1,h j ) (differences between encoder and decoder hidden states), which are used to calculate Attention Weights ij exp(eij )
T kx1ijh j , by softmax function.
Context Vector ci Tjx1ijh j , is calculated for each time step of Decoding by combining Attention Weights with the previous decoder outputs to be passed to decoder RNN ts s1 exp(score(ht , hs )) S exp(score(ht , hs )) .
Although amazingly, such a simple and generic architecture as bidirectional LSTM
with attention (just a few equations and few tens lines of code) can predict (translate,
classify) with such a great result this architecture admits mistakes and has bottlenecks
[
Additionally if to analyze the NLP area not only through the prism of translation where most time machines just translate sentence by sentence and focus on Natural Language Understanding area RNNs do not show good results in overall context understanding and modeling, especially during text generation tasks. This is exactly where the architecture of the transformer can do better. 5.5
In the paper [
The main novelty was that to build a language model there is no need for any recurrent (RNN) or convolutional (CNN) layers at all. Solely self-attention and feed-forward layers are enough.
Encoder
Feed Forward Self-Attention
Decoder
Feed Forward Encoder-Decoder Attention
Self-Attention Fig. 3. Encoder-decoder architecture applied in the transformer.
The transformer includes a lot from what was described before: bidirectionality; encoder-decoder architecture, to support the different length of input-output; self-attention in addition to attention, researchers develop the idea of attention and presented self-attention in the transformer; parallelization (at least on Feed-Forward step which is most expensive) by completely replacing sequential computation (RNNs or Convolutional based) to Attention-based network.
The main components and concepts of this architecture will be presented and described below.
Encoder. The encoder consists of multiple stacked Self-Attention and Feed Forward layers with Residual Connections and Positional Encoder. As usual, the embedding layer is applied in the bottom to convert input sequence to numerical representation. Feed Forward Network does not have dependencies and thus can be parallelized. This is an important concept behind the transformer possibility to learn on a truly big amount of data that LSTM and GRU cannot afford.
Self-Attention Layers help to understand the model, which parts of the input sequence (words) to focus on while encoding sequence. The most important novelty is using three vectors: Query vector, Key vector, and Value vector to create “query,” “key,” and “value” projection of each word in the input sentence.
Decoder. The decoder also consists of multiple (equal to the encoder) stacked SelfAttention, Feed Forward layers with Residual Connections, and additionally encoderdecoder attention layer in the middle. In comparison to the encoder, Decoder’s SelfAttention layer differs. The main idea here is Masking Future Positions. In the encoder, each position can attend to all positions, but in the decoder to prevent leftward information flow to preserve the auto-regressive property, each position can attend only to early positions in the output sequence.
Another important layer of the decoder is Encoder-Decoder Attention layer which gets outputs of the last Attention layer of the encoder as an input and uses Key and Value attention vectors to focus on appropriate places in the input sequence. 6
Embedding. Almost everyone nowadays uses some kind of embedding technique to tokenize input sequences. If your task is not domain-specific, you will probably end up using one of the pre-trained embeddings with dimensionality (300–512). And if you use domain-specific tasks, the choice would be simply based on available computational resources. Starting from simple Word2Vec and Glove and moving to advanced Contextual embedding techniques and increased dimensionality can solve your tasks with high accuracy.
Architecture. As of today, the transformer is the most powerful architecture, which can be trained on enormous amounts of training data with billions of parameters. It is clear that it is impractical to train such a big network from scratch every time, and for every particular task (even today, it will cost hundreds of thousands of USD and enormous computational GPU power). Hence, such a big model comes as pre-trained models, which can then be fine-tuned for various scenarios and tasks. It can be achieved by an additional layer of neurons on the end that was not trained in the pre-training and train them as a part of the new model for specific tasks.
A key advantage of models built using the transformer architecture is that it does not need to be trained with labeled data, so it can learn using any cleaned raw text. This provides the possibility to work with very big datasets and leads to even better accuracy.
The current leaderboard in different NLP competition more and more narrowing to big tech corporations and not universities. That is because of the availability of computational power. On the other hand, leaderboard results and practical implementation is different. Even today, many production-based services use RNN (LSTM and GRU modifications) with attention e.g., for intent classification in widespread chatbot frameworks.
In Fig 4. proposed algorithm for choosing modern approach to solve business NLP tasks considering domain specification of task and availability of resources. Also, it is a good idea to compare BLSTM with Attention and Transformer based architectures results in terms of accuracy, consumption of resources, time to train (hence to improve), interpretability. Significant progress was made in the last decade in the NLP area. Application of deep learning changed rules and uncovered new possibilities with RNNs, BLSTMs, and Attention. Architectures based on Transformers advances even further and show state-ofthe-art results on most of the NLP tasks.
The introduction of deep pre-trained language models in the last couple of year’s significant shift to transfer learning in NLP. At the same time, many of the latest approaches are too demanding in computational resources and algorithms for choosing the right model for the business task presented in current work.
The authors see demand in the sentence boundary detection task. They will research the current area in nearest future using the algorithm provided in the current work to choose models and compare obtained results.