In this paper we research two modifications of recurrent neural networks - Long Short-Term Memory networks and networks with Gated Recurrent Unit with the addition of an attention mechanism to both networks, as well as the Transformer model in the task of generating queries to search engines. GPT-2 by OpenAI was used as the Transformer, which was trained on user queries. Latentsemantic analysis was carried out to identify semantic similarities between the corpus of user queries and queries generated by neural networks. The corpus was converted into a bag of words format, the TFIDF model was applied to it, and a singular value decomposition was performed. Semantic similarity was calculated based on the cosine measure. Also, for a more complete evaluation of the applicability of the models to the task, an expert analysis was carried out to assess the coherence of words in artificially created queries.
Natural language generation is a process of creating meaningful phrases and sentences
in the form of natural language. Two main approaches can be distinguished among the
algorithms for creating texts: methods based on rules and methods based on machine
learning. The first approach allows to achieve high quality texts, but requires
knowledge of the rules of the language and is time consuming to develop [
Currently, generation of texts using neural networks is being actively researched;
one of the most popular algorithms is recurrent neural networks [
The purpose of this article is to study the above-mentioned architectures, analyze their quality and applicability to this task. The use of automatically generated queries to search engines is relevant, since most companies do not issue their search queries for free, and a search engine must be tested while being developed. Also, the received queries can be used to improve the efficiency and optimize the search engine.
Search queries from users of AOL (America Online), which were anonymously
posted on the Internet in 2006, were used in this paper. Although the company did not
identify its users, personal information was present in many queries [
Natural language text generation algorithms are actively studied and used in many software systems, so at the moment there is a large amount of research in this area.
One of the first approaches is the fill-in-the-gap template system. It is used in texts
that have a predefined structure and, if it is necessary to fill in a small amount of data,
this approach can automatically fill in the blanks with data obtained from spreadsheets,
databases, etc. An example of this approach is Microsoft Word mailmerge [
The second step was to add general-purpose programming languages to the first approach that support complex conditionals, loops, etc. This approach is more powerful and useful, but the lack of language capabilities makes it difficult to create systems that can generate quality texts.
The next step in the development of template-based systems is the addition of
wordlevel grammatical functions that deal with morphology and spelling. Such functions
greatly simplify the creation of grammatically correct texts. Next, systems dynamically
create sentences from representations of the values they need to convey. This means
that systems can handle unusual cases without the need to explicitly write code for each
case, and are significantly better at generating high-quality "micro-level" writing.
Finally, in the next stage of development, systems can generate well-structured
documents that are relevant to users. For example, a text that needs to be persuasive can be
based on models of argumentation and behavior change [
After moving from templates to dynamic text generation, it took a long time to
achieve satisfactory results. If we consider the generation of texts in natural language a
subsection of natural language processing, then there is a number of the most developed
algorithms – Markov chains [
Also, the use of generative adversarial networks for text generation is currently being
researched, since they show excellent results in the task of generating images [
User queries in English from the 2006 AOL search engine were selected as data for training neural networks. Researchers try to avoid using this data in their work, as it can be considered revealing, but this paper uses only the texts of the requests themselves, without the IDs of users and websites, that is, without using personal information. The initial data are presented in the form shown in Fig. 1.
Queries longer than 32 words and erroneous requests containing no information were removed from the corpus. Duplicate queries and queries containing website names have also been removed, as they are not natural language examples. In total, 100 thousand queries were randomly selected for training. Fig. 2 shows examples of data after preprocessing.
The queries were separated into character-tokens, each character was assigned with a natural number, the entire corpus was encoded using this dictionary. 3
Recurrent neural networks (RNN) are a family of neural networks where the
connections between the elements form a directed sequence [
However, recurrent networks learn slowly, and their ability to memorize long
dependencies is limited due to the vanishing gradient problem [
Two types of recurrent networks were implemented; they are most often used in the
task of generating texts in natural language – Long Short-Term Memory Network [
Long short-term memory network (LSTM) is a deep learning system that avoids the
vanishing and exploding gradient problems [
Each cell has 3 gates: input, output and forget gates. The forget gate vector is calculated as:
= ( + ℎ −1 + ), where is the input vector, ℎ −1 is the output vector of the previous cell, σ is sigmoid function, , , are weight matrices and bias vector.
= ° −1 + °ĉ , where −1 is the state of the previous cell. Finally, an output vector decides what the next hidden state should be = ( + ℎ −1 + ),
ℎ = °tanh( ).
The results are passed to the next cell. 3.2
The second implemented model is a network with Gated Recurrent Unit (GRU), which
is a new generation of recurrent neural networks, similar to a long short-term memory
network [
Update gate acts as input and forget gates in LSTM and is calculated as:
ℎ = °ℎ −1 + (1 − )° ( ℎ + ℎ( °ℎ −1) + ℎ).
Attention mechanism is a technique used in neural networks to identify dependencies
between parts of input and output data [
Attention mechanism allows a model to determine the importance of each word for
the prediction task by weighing them when creating the text representation. The
approach with a single parameter per input channel was used [
= ℎ , = exp( ) ∑ =1 exp( )
, = ∑ ℎ .
Here ℎ is the representation of the word at time step t, is the weight matrix for the attention layer, are the attention importance scores for each time step, v is the representation vector for the text. 4
All neural networks were implemented in Python 3.7 in Google Collab [
The general architecture of the model is shown in Fig. 5.
The preprocessed data was divided into training and validation sets, which
constituted 80% and 20% of the corpus, respectively. The training data is fed into the
Embedding layer, which converts numbers into vectors that reflect the correspondence
between the character sequences and the projections of those sequences. The resulting
representations are input to the first LSTM layer (GRU), its output is passed to the
second LSTM layer (GRU), and the third in the same way. Next, the output data from
the Embedding layer and these three layers are combined and fed to the
AttentionWeightedAverage layer. The representation vector obtained from the attention layer is
a high-level encoding of the entire text, which is used as input to the final fully
connected layer with Softmax activation for classification [
To check how well the model has trained for a particular epoch, the Categorical CrossEntropy loss function is calculated as
( , ̂) = − ∑ =0 =0( log( ̂ )),
∑ where ̂ are the predicted values.
We conducted experiments with changing the number of LSTM layers (GRU) in the model (2 and 3 layers), as well as adding a Dropout layer after the Embedding layer, which randomly excludes a given number of neurons to prevent network from overfitting and to generalize the model better. Networks with 3 recurrent layers and Dropout performed better.
Bidirectional models of these networks were also trained. Bidirectional recurrent
neural network is a model proposed in 1997 by Mike Schuster and Kuldip Paliwal [
In the case of the generating search queries task, the bidirectional model has shown itself to be better than the unidirectional one; the obtained values of the loss function after training the models for 30 epochs are shown in Table 1.
The value of the loss function decreased on the training data, however, on the validation data, the improvement was less significant, which suggests that the bidirectional model in this task does not learn so well but rather “remembers” the sequences of symbols.
With the help of the implemented model, queries with different "temperatures" were generated. This is a parameter that affects the chance of choosing an unlikely character. 5
Transformer is a deep learning model, which was introduced in 2017 [
Transformers consist of stacks of equal numbers of encoders and decoders. Encoders process input sequences and encode data to show information about them and their characteristics. Decoders do the opposite, they process the information received from the encoder and generate output sequences. All encoders have the same structure and consist of two layers: self-attention and feed-forward neural network. The input sequence being fed into the encoder first passes through the layer of internal attention, which helps the encoder to look at other words in the input sentence while encoding a particular word. The output of this layer is sent to the feedforward neural network. The same network is applied independently to each word. The decoder also contains these two layers, but in between there is an extra layer of attention that allows the decoder to identify the relevant parts of the input sentence.
Internal attention allows the model to see dependencies between the word being processed and other words in the input sequence, which help to better encode the word.
After all decoders, a fully connected Softmax layer is used, which converts the
obtained values into probabilities, from which the largest value is then selected, and the
word corresponding to it becomes the output for this time step.
GPT-2 is a large language model based on Transformer, created by the non-profit
company OpenAI, with parameters ranging from 117 million to 1.5 billion, trained on a
dataset of 8 million web pages [
GPT-2 is built using only decoder blocks, which have the same structure as the Transformer model described above.
GPT-2 does not use words as input but tokens obtained using the Byte Pair Encoding
(BPE) method. It is a data compression technique in which the most common pairs of
consecutive bytes of words are replaced by bytes that do not appear in those words [
Internal attention in GPT-2 also uses masking, which blocks information from tokens to the right of the position that is being calculated. 5.2
A medium size GPT-2 model with 345 million parameters was used, consisting of 24 decoder blocks.
The model was further trained using fine-tuning on the corpus of search queries in English which were also used to train recurrent neural networks. Using the resulting model search queries were generated.
We used the model implementation available at https://github.com/nshepperd/gpt-2. The model was trained for 1000 steps. 6
Latent Semantic Analysis (LSA) is a natural language processing technique for analyzing dependencies between collections of documents and the terms they contain [24].
This method uses a term document matrix that describes the frequency of occurrence
of terms in a collection of documents. The elements of such a matrix can be weighted,
for example, using TF-IDF: the weight of each element of the matrix is proportional to
the number of times the term occurs in each document, and inversely proportional to
the number of times the term occurs in all documents in the collection. After compiling
the term-document matrix, its singular value decomposition is carried out, i.e. it is
represented as = , where matrices U and V are orthogonal, and S is a diagonal
matrix, the values of which are called singular values of matrix A. This expansion
reflects the basic structure of dependencies present in the original matrix, allowing to
ignore noise [
To carry out latent semantic analysis, the gensim library for Python was used [
Comparing each document, in this case a query, with documents from the corpus with real queries, the method returns a value from -1 to 1, reflecting the semantic similarity of the documents. The analysis results are shown in Table 2. The corpus of real queries is varied, so the average value of the result of comparing each generated document with all documents from the reference corpus differs slightly from zero. At the same time, for each query artificially created using GRU and LSTM networks, there are on average 16 and 14 semantically close documents, when the values are greater than 0.7, and for the GPT-2 model, this number is 9 documents. Also, for each request generated by the GPT-2 model, out of 10,000 compared documents, 2684 have a value greater than 0, and for LSTM and GRU networks, 4659 and 4000, respectively. From this, we can conclude that LSTM and GRU used more words semantically similar to words from the training data when generating queries than GPT2. This makes sense, since the first two models were trained from scratch on the input data, while the main training of the last model took place on a completely different corpus, it was only fine-tuned in order to generate queries suitable for structure. It is also important to take into account that the comparison was carried out with 10 thousand reference queries, although the models were trained on 100 thousand, therefore not all dependencies were taken into account, however, the obtained values are sufficient for analysis.
The analysis results show that the generated queries have similar semantics to the corpus of real user queries, but at the same time they do not repeat them literally, that is, they are new queries in meaning.
The GRU and LSTM networks were trained by characters and could have generated non-existent words, so it was decided to test them. Each word from the queries was checked for existence using a corpus containing more than 466 thousand English words, available at https://github.com/dwyl/english-words. In the queries generated by the GRU network, 141 words out of 4431 were not found, and in the queries of the LSTM model - 166 out of 4325. The words that were not found contained typos or mistakes in words that the models remembered. Therefore, it may be worth preprocessing the data by correcting typos and errors of this kind. However, queries with typos can be useful depending on the task in which they will be applied. So, for example, when they are used to test a new search engine or to optimize it, they will be more relevant with typos, as they have a greater similarity with real user queries.
Due to the fact that neural networks cannot understand the meaning of a sentence, although they often find the correct dependencies between tokens, an expert (manual) analysis was carried out to assess the quality of the generated search queries.
From the queries generated by each model, 100 queries were randomly selected. It was determined whether each search query makes sense, whether it is similar to a real possible user query. It should be noted that this assessment is subjective. Queries were considered "good" if the words in them were consistent with each other.
The analysis results are shown in Table 3. The table shows that the GRU and LSTM networks showed almost the same results, while GPT-2 is slightly better. During the analysis, it was observed that the GPT-2 model generates shorter queries than the other two models.
The results of the analysis showed that the GRU and LSTM networks have approximately the same quality when solving the task of generating search queries, and the GPT-2 model was worse in automatic analysis, but better in expert judgment. Therefore, this model is better suited for generating search queries, since the significance of the expert judgment is higher than the automatic one, although for more accurate results it is worth carrying out this assessment with the help of other experts. 8
In the course of this work, we researched the leading models used to generate texts in natural language and their ability to solve the task of generating queries for search engines and we conducted their comparative analysis. Two neural networks are fully implemented: a network with a long short-term memory and a network with a gated recurrent unit. The GPT-2 architecture based on the Transformer model was researched; it was also fine-tuned using the corpus of real user requests.
Latent semantic analysis showed that the GPT-2 model performs worse than the other two networks. However, the automatic metrics for evaluating the generated text do not always reflect the quality of the model, since at the moment it is impossible to assess the meaningfulness of the texts using the algorithm. To solve this problem, an expert analysis of the generated texts was also carried out, according to the results of which the GPT-2 model was better than the other two models. At the same time, the LSTM and GRU networks showed approximately the same quality according to the results of all analyses performed.
Acknowledgments. This work was subsidy of the Russian fund of fundamental research, grant agreement 18-07-00964.