Recent years have witnessed significant advancements in neural text generation driven by the emergence of large language models and growing interest in this field. This systematic review aims to identify and summarize current trends, approaches, and methods in neural text generation from 2022 to 2024, complementing the findings of a previous review covering 2015-2021. Following the PRISMA methodology, 43 articles were selected from the Scopus database for analysis. The review reveals a shift towards innovative model architectures like Transformerbased models (GPT-2, GPT-3, BERT), attention mechanisms, and controllable text generation. While BLEU, ROUGE, and human evaluation remain the most popular evaluation metrics, new metrics like BERTScore have emerged. Datasets span diverse domains and data types, with growing interest in unlabeled data. Applications have expanded to areas such as table-to-text generation, knowledge graph-based generation, and medical text generation. Although English dominates, there is increasing research on low-resource languages. The findings highlight the rapid evolution of neural text generation methods, the broadening of application areas, and promising avenues for future research.
Analysis
Generation n o i t a z i r a m m u s t x e T g n i r e w s n a n o i t s e u Q n o i t a l s n a r t e n i h c a M g n i l l e d o m e g a u g n a L n o i t a r e n e g e g a u g n a l l a r u t a N 2) by quality metrics: – LSTM – Long Short-Term Memory network, works better than RNN for larger data volumes; – GRU – Gated Recurrent Unit (simplified version of LSTM); – CNN – Convolutional Neural Network. • innovative: – Attention Based – networks that use an attention mechanism to increase the importance of input data; – Transformer – networks that use an attention mechanism without recurrent or convolutional layers; – BERT – a neural network developed by Google that combines attention mechanisms without recurrent or convolutional layers with bidirectional encoders • human-centered: – Domain-Expert – involving a person who is an expert in the given field to validate the results. • machine-centered (automatic): – BLEU (bilingual evaluation understudy) – compares the number and value of tokens (lexemes) of machine and human translation; the meaning of words is not taken into account; – ROUGE (Recall-Oriented Understudy for Gisting Evaluation) – compares machine-generated and human-generated summaries/translations; – Cosine Similarity – comparison of the cosines of the angle of two non-zero vectors: a value of +1 corresponds to unidirectional proportional vectors, -1 corresponds to oppositely directed proportional vectors; – Content Selection – a metric similar to ROUGE that uses an attention mechanism for a given task; – Diversity Score – a metric for evaluating diversity. 3) by application of the neural network: • AMR (Abstract Meaning Representation) – extracting semantic relationships from text; • Language Generation – generating human-like text; • Speech-to-text – converting speech to text; • Script Generation – generating scripts based on given words; • Machine Translation – generating machine translation of text from one language to another; • Text Summarization – generating a summary for a given text; • Image Captioning – generating a description for a given image; • Shopping Guide – generating an advertising description for a given product image; • Weather Forecast – generating a weather forecast text. 4) by generation language: • well-resourced: English, Chinese; • low-resourced: Bengali, Korean, Balinese, Spanish, Hindi, Slovak, Macedonian. 5) by dataset for training the neural network: • by annotation type: – Labeled (labeled data); – Unlabeled (unlabeled data); • by type: – Sentence – sentence; – Paragraph – paragraph; – Question/answer – question and answer type data; – Document – document type data.
To obtain the results presented in figure 3, Fatima et al. [
Similarly, the research questions were refined: RQ1. What advanced deep learning methods are used for text generation in the literature of 2022-2024? RQ2. What new metrics for evaluating the efectiveness of text generation models are there in the literature of 2022-2024? RQ3. What text generation datasets are described in the literature of 2022-2024? RQ4. What new text generation applications are described in the literature of 2022-2024? RQ5. What natural languages are used for text generation in the literature of 2022-2024?
Systematic literature analysis is the main method of this research, which allows generalising and
synthesising information from a large number of scientific publications (secondary sources) according
to a clearly defined methodology. The PRISMA (Preferred Reporting Items for Systematic Reviews
and Meta-Analyses) methodology, which is a generally recognised standard for systematic reviews
and meta-analyses in various fields of science, was chosen for conducting the review [
The chosen method fully corresponds to the aim and objectives of the research, allowing to obtain a generalised picture of the current state of research in the field of text content generation based on the analysis of a significant array of scientific publications in recent years.
Fatima et al. [
Currently, Scopus covers about 90% of IEEE Xplore and ACM Digital Library, Web of Science – about 50%; ScienceDirect and Scopus have the same owner – Elsevier. Given that Scopus includes a significant part of these libraries, only one database – Scopus – was used instead of 2 databases and 4 libraries. Applying the search query from the previous review (Table 1) yields 2580 documents for 2015–2020 (versus 100 documents specified in [ 2, p. 53494]). When searching only in article titles, the number of documents decreases to 109 and there is a partial match with the list of sources [2, p. 53500-53503]).
The inability to reproduce the previous results for the query from table 1 prompted the creation of a new query: (
TITLE-ABS-KEY(neural network) OR TITLE-ABS-KEY(machine learning) OR TITLE-ABS-KEY(deep learning) ) AND
TITLE("text generation") Inclusion criteria: Exclusion criteria:
The first part of the query was simplified to three key phrases, two of which (“neural network” and “deep learning”) match the first group of table 1, and the third (“machine learning”) generalizes all other keywords of the first group, including those that did not exist at the time of the previous review. The second part of the query included only the key phrase “text generation”, the search for which is performed in document titles (TITLE), and not in titles, abstracts and author keywords (TITLE-ABS-KEY). 2.2. Document inclusion and exclusion criteria 1. Documents published between 2022 and 2024. 2. Documents related to text generation using artificial neural networks. 3. Documents describing approaches, architectures, quality metrics, languages, datasets or applications of text generation. 1. Documents published before 2022 or those that do not contain data for 2022-2024. 2. Documents that are not related to text generation or do not use artificial neural networks. 3. Documents that do not contain relevant information regarding the posed research questions (new methods, metrics, datasets, applications, natural languages).
The Scopus query on 04.03.2024 returned 248 documents, the distribution of which by year is shown in ifgure 4. Of these, 2 were duplicates and 157 were dated before 2022, so they were excluded from the list for obtaining.
Figure 5 presents a scheme of data selection for the systematic review.
An attempt was made to obtain 89 documents from publishers’ websites, the scientific social network ResearchGate, and preprint servers (primarily arXiv). 41 documents (primarily from the ACM Digital Library and IEEE Xplore) could not be obtained. Thus, 48 documents were selected for evaluation, the review of which revealed 1 document that did not contain data for 2022-2024, and 4 documents that did not contain relevant information regarding the posed research questions.
43 documents were selected for review: [
Characteristics of the used datasets - data type: sentence, paragraph, document, question-answer, not specified Characteristics of the used datasets - size Characteristics of the used datasets - format: CSV, JSON, XML, files, not specified Characteristics of the used datasets - by annotation type: labeled data, unlabeled data Characteristics of the used datasets - data quality: raw (unprocessed), preprocessed Characteristics of the used datasets - by availability: publicly available, private, not specified Characteristics of the used datasets - link Solved text generation task (what was the neural network used for) Language of text generation
An example of a response is shown in figure 6.
For external verification, the LLM GPT 4 was used through the Scholar GPT [
Comparison of the description of the article [
Records identified from
Scopus (n = 248) Records screened
(n = 246) Documents sought for retrieval (n = 89) Documents assessed for eligibility (n = 48) Documents included in review (n = 43)
Records removed before screening: duplicate records removed (n = 2)
Excluded records before 2022 (n = 157) Documents not retrieved (n = 41)
Excluded documents: a) do not contain data for
2022-2024 (n = 1); b) do not contain relevant information regarding the posed research questions (n = 4).
Search and learning for unsupervised text Search and Learning for Unsupervised Text generation Generation 2022 2022
Continued on next page
Table 2 – continued from previous page
Query Claude 3 Sonnet Response Countries represented by Canada the authors
GPT 4 Response Canada (The author, Lili Mou, is afiliated with the University of Alberta in Edmonton, Canada) Article purpose To introduce a search and learning ap- The article aims to introduce approaches to proach for unweighted text generation, unsupervised text generation using search where the task of text generation is formu- and learning methodologies. It focuses on lated as a search problem with a heuristic overcoming the need for massive labeled function for evaluating the quality of a can- datasets, which are required in supervised didate sentence, and discrete local search al- methods, thereby making text generation gorithms are used to maximize this evalua- feasible for low-resource languages and retion function. A machine learning model is ducing the human labor required for data also trained on the search results to smooth annotation.
out noise and improve eficiency.
Used neural network ar- Recurrent Neural Networks, Transformers, Neural architectures mentioned include rechitectures in particular GPT-2 current neural networks (RNNs), Transformer architecture, and language models such as RoBERTa and pretrained language models by Radford et al., 2019.
Used quality metrics iBLEU (for evaluating paraphrasing) Metrics such as language fluency, semantic coherency, and task-specific constraints (like sentence simplicity scores) are used to evaluate the quality of generated texts.
Characteristics of the used datasets - name Quora dataset (for the task of paraphrasing). Not specified in detail for individual
No names specified for other tasks datasets. - data type: sentence, sentence, paragraph (for the task of sum- Sentences, as the text generation involves paragraph, document, marization) tasks like paraphrasing, summarization, question-answer, not etc. specified - size Not specified Specific sizes are not mentioned, but the use of large-scale generic corpora is implied. - format: CSV, JSON, Not specified Not explicitly mentioned, but it includes laXML, files, not specified beled input-output pairs in various formats typical for NLP tasks. - by annotation type: la- unlabeled data Primarily deals with labeled data in superbeled data, unlabeled vised contexts, though the focus here is on data reducing dependence on such datasets. - data quality: raw (un- Not specified The discussion includes the use of both raw processed), preprocessed and preprocessed data in diferent contexts. - publicly available, pri- Not specified The text suggests a shift towards methods vate, not specified that do not require publicly available or parallel datasets. - link Not specified No specific dataset URLs provided. Solved text generation Paraphrasing, summarization, text simplifi- The neural networks are applied to genertask (what was the neu- cation, sentence generation from keywords, ate text for various applications including ral network used for) grammar error correction, style transfer paraphrasing, summarization, text simpliifcation, and creating sentences from keywords, all in an unsupervised manner.
Language of text genera- English The tasks are primarily demonstrated with tion English examples, though the methodologies are applicable to multiple languages.
Thus, LLMs can accurately extract key information from articles, although they may sometimes omit details that are not explicitly stated in the text. To minimize the risk of incorrect information extraction, a human review of Claude 3 Sonnet’s results was performed. To avoid issues related to terminology translation, LLM responses were additionally requested in the language of the selected documents (English).
To assess the quality of the research selection and analysis process in this review, the following criteria were applied: 1. Clarity and relevance of the research inclusion and exclusion criteria to the purpose of the review. 2. Completeness and systematic nature of the search for relevant research in the selected databases. 3. Consistency and reproducibility of the research selection process according to the inclusion and exclusion criteria. 4. Application of a standardized review map for collecting and systematizing data from selected studies. 5. Involvement of at least two independent researchers in the process of data selection, analysis, and synthesis to minimize the risk of bias. 6. Consideration and description of any discrepancies or uncertainties in the process of research selection and analysis. 7. Ensuring transparency and reproducibility of the review process by detailed description of each stage in the report.
Adherence to these quality criteria made it possible to ensure the reliability and validity of the results and conclusions of this systematic review.
PRISMA provides for the presence of the following additional components in the research methodology: • assessment of the risk of bias in the selected studies is not relevant because this review considers diferent approaches and methods of text generation, and does not compare the results of individual studies; • determination of the efect size for each outcome (or type of outcome) is not performed because this review does not aim to conduct a meta-analysis or quantitative synthesis of the results; • description of the methods of synthesizing research results, such as meta-analysis, is not performed because the review does not involve a quantitative synthesis of the results; • assessment of the risk of bias due to incomplete presentation of the results in publications is not performed because this review focuses on describing and classifying existing approaches and methods. • assessments of the reliability and trustworthiness of the results obtained from publications are not performed due to the use of reliable sources: publications selected by Scopus.
In [
As can be seen from figure 7, the number of articles in journals (ARTICLE) prevails over the number of conference proceedings articles (CONFERENCE) during 2022-2024. In 2022, the number of conference proceedings documents (15) was significantly higher than the number of articles in journals (4), but in 2023 there is an increase in the number of articles in journals (16) compared to conference proceedings articles (6). For January and February 2024, there are only articles in journals (2), and conference proceedings articles are absent. In total, for the period 2022-2024, the number of articles in journals 2 0 2024 (22) is equal to the number of conference proceedings articles (21). The increase may indicate a more thorough coverage of the issue in scientific journals compared to conference proceedings in recent years.
CONFERENCE ARTICLE 3.2. RQ1: What advanced deep learning methods are used for text generation in the literature of 2022-2024? Table 3 presents an overview of neural network architectures used for text generation, according to data from 2022-2024 studies.
GRU (Gated Recurrent A simplified variant of LSTM with – Unit) fewer parameters.
CNN (Convolutional Convolutional neural networks, often YOLOv5 Neural Networks) used for image processing.
Graph Neural Networks Models that work with graph data GraphWriter, CGE-LW [
Table 3 – continued from previous page
Architecture Description Representatives Articles
Transformer Architecture that uses an attention T5, CodeT5, TrICY, [7, 9, 18, 19, 31,
mechanism for processing sequential DETR 47, 8, 20, 32,
data. 22, 27, 41, 39,
43, 34, 44, 19,
48]
BERT (Bidirectional A Transformer-based model trained PubmedBERT, Bi- [8, 37, 13, 18,
Encoder Representations on large amounts of unlabeled text. oLinkBERT, RoBERTa, 19, 20, 26, 28,
from Transformers) XLM-RoBERTa 30, 35, 32, 12,
9, 39, 40, 45]
GPT-2, GPT-3 (Genera- Transformer-based models used for OPT, Llama, CodeBERT [6, 8, 10, 11, 13,
tive Pre-trained Trans- text generation. 47, 15, 18, 21,
former) 22, 12, 23, 24,
26, 19, 33, 25,
32, 37, 34, 45,
36, 39, 43, 44]
– [
Seq2Seq (Sequence-to- Architecture that uses an encoder and S2ST, S2SL, S2SG, [39, 15, 42, 28,
Sequence) decoder to generate sequences. S2ST+, D+ Full, DSG 31, 46, 43]
GAN (Generative Adver-Generative adversarial networks con-EGAN, TILGAN, [
DGSAN
Memory Networks Models that use external memory for DM-NLG (with [
Mem2Seq, GLMP
Difusion Models Models that use a difusion process to GENIE, NAT, iNAT, [
ProphetNet, InsT,
CMLM, LevT, BANG,
ConstLeven
Prompt-based models – [
Table 4 presents a summary of text generation approaches based on the data from table 3.
Among the innovative approaches, the most popular are the use of models based on the Transformer architecture, in particular GPT-2, GPT-3, BERT and their variations. These models demonstrate high efifciency in generating coherent and semantically relevant text. Approaches using attention mechanisms and controllable text generation are also gaining popularity.
Traditional approaches, although used less frequently, still find their application in certain tasks, such as image-based text generation, machine translation and others.
Overall, there is a trend towards the transition from traditional approaches to more innovative and eficient models based on the Transformer architecture and attention mechanisms. This allows improving the quality of the generated text and expanding the scope of application of these technologies.
Figure 8 shows that in 2022 and 2023, innovative approaches to text generation prevail, while traditional approaches and a combination of approaches are less common. In 2024, there are articles that use innovative and combined approaches in equal numbers, but the sample for this year is incomplete, since data were collected only for part of the year. In general, there is a trend towards an increase in the number of studies applying innovative approaches, such as models based on the Transformer architecture and attention mechanisms.
• Traditional approaches, such as RNN, LSTM, CNN, are still used for text generation, but to a lesser extent compared to innovative approaches. • The Transformer architecture and its variants (GPT-2, GPT-3, BERT) have gained significant popularity in 2022-2024, demonstrating high eficiency in generating coherent and semantically relevant text. • New architectures and approaches have emerged, such as Difusion Models and Memory Networks models, which were not presented in the previous review. • Considerable attention is paid to models that use attention mechanisms and controllable text generation. • There is a trend towards combining traditional and innovative approaches to achieve better results in text generation. • Overall, in 2022-2024, there is a transition from traditional approaches to more innovative and eficient models based on the Transformer architecture and attention mechanisms, which allows improving the quality of generated text and expanding the scope of application of these technologies.
Traditional approaches Innovative approaches Combination of approaches
Comparing the obtained results with the data from the previous systematic review [
Thus, comparing the results of the two reviews demonstrates that although traditional metrics, such as BLEU and ROUGE, remain widely used, new metrics appear in 2022-2024 that take into account various aspects of generated text quality. This indicates the active development of quality assessment methods and the search for more efective and informative approaches to evaluating text generation models.
A metric that evaluates the quality of machine translation.
A metric that evaluates the quality of generated text using a pre-trained BERT model.
A metric that evaluates the quality of automatic image captioning by comparing machine-generated captions with sets of reference captions.
A metric that evaluates the quality of a language model.
A metric that evaluates the quality of
classification, particularly in binary
classification tasks.
[
A metric that evaluates the diversity of Dist-1, Dist-2, Dist-3, [
Table 6 provides an overview of the quality evaluation metrics applied in the articles. Most studies use machine-centred metrics for automatic evaluation of generated text quality. A significantly smaller number of studies apply human-centred metrics, which may be due to the labour-intensive and subjective nature of human quality assessment. However, the use of human-centred metrics remains important for obtaining a more complete and reliable evaluation of text generation quality. Some studies do not apply any quality metrics, which may be related to the focus on other aspects of text generation, such as eficiency or speed of model operation.
Machine-centred [
The use of diverse quality metrics is important for a comprehensive evaluation of the efectiveness of models and approaches to text generation. Combining machine-centred and human-centred metrics allows obtaining more reliable and valid evaluation results.
The diagram in figure 9 shows that the most frequently used quality metrics are BLEU (55.8% of articles) and ROUGE (48.8% of articles). Human Evaluation is also quite common – it is applied in 23.3% of articles. Other metrics, such as Perplexity, METEOR, BERTScore, and Distinct-n, are used less frequently but still have a significant share of mentions in articles. The least common metrics are the Turing Test, Fluency, Coherence, Diversity, N-gram Overlap, and Embedding Similarity, each of which is mentioned in only one article (2.3%).
Automatic quality metrics, such as BLEU and ROUGE, are the most widely used for evaluating the efectiveness of text generation models, while human quality evaluation is used less frequently but remains an important component for obtaining a more complete and reliable assessment of generated text quality.
Comparing the obtained results with the data from the previous systematic review [
Thus, comparing the results of the two reviews demonstrates that while traditional metrics, such as BLEU and ROUGE, remain widely used, new metrics appear in 2022-2024 that take into account various aspects of generated text quality. This indicates the active development of quality assessment methods and the search for more efective and informative approaches to evaluating text generation models. Table 7 – continued from previous page
Dataset name Custom dataset for Bangla word sign language Synthetic dataset Penn Treebank IWSLT’14 De-En WMT16 English-German WMT17 English-German WMT20 WMT21 WMT’14 German-English Multi-News Java Python English ATIS ViGGO TREC Korean Weather Rest KLUE-TC C4 M2D2 Political Slant Layof MC M&A Flood Wildfire Boston Bombings Bohol Earthquake West Texas Explosion Dublin New York City WSC CBT-CN CBT-NE Wikihow SAMSum DART Custom dataset composed of tweets labeled with emotions AFQMC CHIP-STS QQP MRPC ParaNMT-small NIST Chinese-English GTZAN Minions Japanimation
The analysed studies use a wide range of datasets covering various domains and types of texts, from user reviews and news articles to medical and technical texts. This indicates the active development and application of text generation methods in diverse fields.
Table 8 presents the data types used in the reviewed articles, sorted in descending order by the number of mentions. Datasets containing sentences are used most often – they are mentioned in 26 articles. In 5 articles, the data type is not explicitly specified. Other data types, such as paragraphs (18 articles), documents (11 articles), question-answer (10 articles), descriptive tables (9 articles), translations (7 articles), stories (4 articles), images (4 articles), and others, are less common.
The prevalence of datasets with sentences may be due to the fact that many text generation tasks, such as machine translation, paraphrasing, question answering, etc., often work at the sentence level. At the same time, the presence of various data types, including paragraphs, documents, images, music, and others, indicates that text generation methods can be applied to a wide range of tasks and domains.
Table 9 presents the data annotation types used in the reviewed articles, sorted in descending order by the number of mentions. Labeled datasets are used most often – they are mentioned in 22 articles. In 20 articles, the annotation type is not explicitly specified. In 5 articles, unlabeled data are used. In 4 articles, both labeled and unlabeled data are used.
The prevalence of labeled datasets may be due to the fact that many text generation tasks, especially those that use controlled approaches or require compliance with certain templates or structures, require labeled data for training models. Annotation can include elements such as parts of speech, syntactic structures, semantic roles, tags for controlled generation, etc.
At the same time, the presence of studies that use unlabeled data or a combination of labeled and unlabeled data indicates the active development of unsupervised and semi-supervised learning methods in the field of text generation. These approaches allow using large volumes of unlabeled text data for
Preprocessed [
Not specified [
Table 10 presents the data quality used in the reviewed articles, sorted in descending order by the number of mentions. In 28 articles, the data quality is not explicitly specified. In 12 articles, preprocessed data are used, while in 10 articles – raw data. In 4 articles, both preprocessed and raw data are used.
Preprocessed data usually go through the stages of cleaning, normalization, tokenization, and sometimes additional annotation before being used in model training. This improves the quality and consistency of the data, as well as facilitates the learning process. Examples of preprocessed data can be datasets obtained from existing corpora or databases that have already undergone some processing.
Raw data, on the other hand, are data obtained directly from real sources, such as web pages, social networks, unprocessed texts, etc. They can contain noise, incorrect formatting, errors, and other artifacts. Using raw data can be useful for training models that need to be robust to real conditions and able to process unstructured data.
The lack of information about data quality in a significant part of the analyzed articles may indicate that the authors do not pay enough attention to this aspect or consider it less important for the research. At the same time, data quality is a critical factor that afects the eficiency and generalizability of text generation models, so it is worth paying more attention to the description and analysis of the quality of the data used in future research.
Comparing the results of the 2022-2024 review with the previous review [
Thus, comparing the results of the two reviews demonstrates that text generation datasets continue to actively develop, covering new domains and data types. At the same time, some popular datasets remain relevant and widely used in research. There is a trend towards the use of more diverse data types and a growing interest in unlabeled data and combined approaches. However, the description of data quality still requires more attention in future research to ensure the reliability and reproducibility of the results. 3.5. RQ4: What new text generation applications are described in the literature of 2022-2024? Table 11 shows the text generation applications found in the analyzed articles, sorted in descending order by the number of references. The most common applications are text summarization (8 articles), machine translation (8 articles), table-to-text generation (5 articles), paraphrasing and data augmentation (4 articles each). Other applications, such as controllable text generation, image-based text generation, text generation from knowledge graphs, etc., are mentioned in a smaller number of articles.
Figure 10 visualizes the text generation applications listed in table 11 as a diagram. The diagram clearly shows the prevalence of table-to-text generation, text generation from knowledge graphs, controllable text generation, and medical text generation applications compared to other areas.
Machine Translation Text Summarization Table-to-Text Generation Data-to-Text Generation Paraphrasing Data Augmentation Controllable Text Generation Image-based Text Generation Text Generation from Knowledge Graphs MIDI or Music File Generation Script Generation News Headline Generation Technical Documentation Generation Cybersecurity Encyclopedic Text Generation Sign Language to Text Translation
The analysis of text generation applications demonstrates a wide range of possibilities for using this technology in various fields, from processing structured data to creating emotionally colored texts and translating sign language into text. The development of new methods and neural network architectures opens up new prospects for further expanding the areas of text generation application.
Comparing the results of the 2022-2024 review with the previous review [
Thus, comparing the results of the two reviews demonstrates that the field of text generation application continues to actively expand, covering new areas and directions. The popularity of such applications as text generation from tables and knowledge graphs, controllable text generation, and medical text generation indicates a growing interest in methods that allow eficiently processing structured data and obtaining more relevant and high-quality results. At the same time, traditional applications, such as paraphrasing, text summarization, and machine translation, remain relevant and widely used in research. 3.6. RQ5: What natural languages are used for text generation in the literature of 2022-2024? Table 12 presents an extended annual summary of the languages used for text generation in 2022-2024. English is the most widely used language, with 38 articles covering all three years. Various neural network architectures are used for generating English texts, including Transformer, BERT, GPT-2, GPT-3, RNN, LSTM, CNN, GAN, and Seq2Seq.
German is represented in 5 articles using GAN architectures (Conditional GAN, StyleGAN, DCGAN). Chinese is represented in 4 articles using Graph Neural Networks and B2T architecture. Bengali is represented in 2 articles (one in 2022 and one in 2023) dedicated to recognition using CNN and YOLO. Romanian is represented in 2 articles (one in 2022 and one in 2023) using DCGAN and BART architectures. French, Urdu, Shakespearean English, and Korean are each mentioned in one article, using various architectures such as Conditional GAN, StyleGAN, DCGAN, and GPT-2.
In 2023, a study by Taunk et al. [
Conditional GAN, StyleGAN, DCGAN Graph Neural Networks, B2T Conditional GAN, StyleGAN, DCGAN CNN, YOLO, mBART
GPT-2
Continued on next page
Table 12 – continued from previous page
2022 2023 2024 Total
– 1 – 1
[
Architectures HipoRank, mBART, mT5 –
Comparing the results of the 2022-2024 review with the previous review [
This indicates a growing interest in developing text generation models for diverse languages.
• The study [
Thus, comparing the results of the two reviews demonstrates that although English remains the dominant language in text generation research, there is a growing interest in developing models for other languages, especially low-resource languages. The emergence of studies dedicated to generating texts in languages such as Urdu, Hindi, Malayalam, Marathi, Oriya, Punjabi, and Tamil indicates an expansion of the possibilities for applying text generation to diverse languages. Furthermore, the use of modern neural network architectures such as Transformer, BERT, GPT-2, GPT-3, GAN, and Graph Neural Networks allows improving the quality and eficiency of text generation for various languages.
Comparing the language distributions in the old and new reviews with the distribution of languages
by the number of models on Hugging Face [
Comparing the language distributions shows that despite the dominance of English in research and available resources, there is a growing interest in text generation in other languages, especially low-resource ones. However, the number of available models and resources for these languages is still significantly lower compared to English. Furthermore, the presence of a large number of models for some languages on Hugging Face that are rarely mentioned in the reviews indicates the potential for further research and development in this field.
The paper presented the results of a systematic review of the application of artificial neural networks
for generating textual content in 2022-2024 and compared them with the results of the previous review
[
Overall, there is a shift from traditional approaches to more innovative and eficient models. 3. Among the metrics for evaluating the efectiveness of text generation models, BLEU and ROUGE are the most widely used, along with human evaluation. In 2022-2024, new metrics such as BERTScore, Fluency, Coherence, Diversity, N-gram Overlap, and Embedding Similarity appeared, indicating active development of methods for assessing the quality of generated text. 4. Datasets for text generation continue to actively develop, covering new domains and types of data. There is a trend towards using more diverse types of data (tables with descriptions, images, music, translations, etc.) and a growing interest in unlabeled data and combined approaches. 5. The field of text generation applications continues to actively expand, covering new areas and directions. The popularity of applications such as text generation from tables and knowledge graphs, controlled text generation, and medical text generation indicates a growing interest in methods that allow eficient processing of structured data and obtaining more relevant and high-quality results. 6. Although English remains the dominant language in text generation research, there is a growing interest in developing models for other languages, especially low-resource languages. The use of modern neural network architectures allows improving the quality and eficiency of text generation for various languages.
The results of this review demonstrate the active development of the field of text generation in 2022-2024, characterized by the emergence of new approaches, metrics, datasets, and the expansion of application areas.
Despite significant progress in the development of text generation technologies, questions remain open regarding the assessment of the quality of generated text, the adaptation of models to diferent subject domains and languages, and the ethical aspects of using these technologies. Further research can be aimed at solving these problems and developing more efective, universal, and safe text generation models.
Declaration on Generative AI: During the preparation of this work, the authors used Claude 3 Opus in order to: Text Translation, Abstract drafting, Formatting assistance. After using this tool, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.
A. Review map for an article 1. Bibliographic reference 2. Document type: journal article or conference paper 3. Title 4. Year of publication 5. Countries represented by the authors 6. Purpose of the article 7. Neural network architectures used 8. Quality metrics used 9. Characteristics of the datasets used 10. Text generation task solved (what the neural network was used for) 11. Language of text generation