The rapid development of generative Artificial Intelligence (AI) has caused an augment of interest in society in AI tools. These tools can produce a positive impact in lots of areas, saving the time and effort of solving some tasks [1,2,3].
In particular, state-of-the-art Natural Language Generation (NLG) tools can produce text that, in some cases, can be indistinguishable from human-generated texts. This could have lots of benefits in some sectors such as academia, tourism or marketing [4]. Nonetheless, these tools also have some drawbacks. First of all, text generated by these tools may contain hallucinations, which is the phenomena that occur when a text is nonsensical or unfaithful to the provided source [5]. Secondly, AI-generated text could be biased in some cases, which is the misrepresentation or attribution errors that result in favouring certain groups or ideas [6]. Finally, these tools also lack of logical reasoning, a fact that it is essential to human intelligence [7]. In the wake of these limitations, these tools can be used in a bad and unethical way to potentially generate dis-and mis-information.
Moreover, the core of these tools are Large Language Models (LLMs). The expense of time and cost needed to train these models is extremely high, being only within the reach of large companies.
Therefore, the motivation for the present research arises from the need in the academia to find efficient architectures that could produce text in a controlled manner, achieving a similar performance to state-of-the-art models, but solving the hallucination issue.
The remainder of this article is organised as follows: Section 2 presents an overview of the relevant literature concerning NLG; Section 3 shows the main hypotheses and objectives planned for this research; Finally, Section 4, and Section 5 detail the methodology this PhD will follow and some relevant research topics for discussion.
Before introducing my proposal, this section aims to contextualise this study within the state of the art of the NLG.
NLG is the subfield in the Natural Language Processing (NLP) area that aims to produce meaningful sentences to meet a communicative goal [8]. Depending on several aspects of the generation, NLG can be classified according to two criteria:
• Type of input: Depending on the type of input, NLG can be catalogued as (1) text-to-text generation (T2T) and ( 2) data-to-text generation (D2T) [9]. In D2T, input data can assume different types such as binary data, images, voice, database, ontologies, etc. Recently, another concept of NLG has emerged, (3) none-to-text generation (N2T) [10], which corresponds to the generation to which no input is received. • Task typology: Based on the communicative goal, NLG can be grouped into (1) text abbreviation;
(2) text expansion; and (3) text rewriting and reasoning. Text abbreviation tasks consist in detecting the most important information in a text and fusing that information into a short text, e.g. text summarisation. Text expansion tasks aim to generate complete sentences from some meaningful words, e.g. topic-to-essay. Finally, Text rewriting and reasoning tasks try to rewrite a text into another style or apply reasoning methods, e.g. text simplification.
To achieve the communicative goal of these tasks, the NLG area has been studied for a long time. First researches date by the end of 1970 [11]. Notwithstanding, it has not been until recent years that the NLG field has achieved an exponential improvement, producing text in a very similar way to humans. But, how did we get to this?
In a first stage, the NLG task was seen as a sequential scheme of four different stages (preprocessing, macroplanning, microplanning and realisation). Modular architectures followed this scheme, making a clear distinction between the distinct sub-tasks of each stage. The most famous modular architecture was proposed by Reiter [12]. Figure 1 shows the sub-task division in this architecture.
Other works within this architecture can be found in [13,14,15,16].
Later, that clear distinction between the distinct sub-tasks became more flexible originating what is known as planning perspectives. This scheme was similar to the employed in modular architectures, but it allows to combine and implement two or more different sub-tasks in one sub-task, e.g. to combine text structuring and sentence aggregation sub-tasks. Some examples of this approach are present in [17,18,19,20,21,22,23,24].
Finally, the sub-task division started to disappear, originating global approaches. This type of architecture does not make a distinction among sub-tasks, performing every task as a whole, and relying on statistical learning and neural networks. Some proposed architectures within global approaches are: Graph Neural Networks [25], Generative Adversarial Nets [26], Recurrent Neural Networks [27], Pre-trained Models [28], Memory Networks [29], Transformers [30] and Copy and Pointing Mechanism [31]. This group of approaches have made the major development in the NLG area. The most important proposal in this group was the Transformers architecture and its concept of attention. Models based on this architecture achieve a high performance at NLG tasks. The best-performing models based on Transformers are LLMs such as GPT4 [32] or LLaMa [33], which have neural networks with billions of parameters. Nowadays, most of the research in the industry is focused on developing bigger LLMs, as it is thought that a bigger LLM would achieve better performance. The cost and time of training these models are unassumable for the academia. On account of that issue, there is a need in the academia to find more efficient architectures that could perform similarly to LLMs.
Consequently, my line of work will focus on exploring efficient architectures that could generate text with similar results to state-of-the-art models. Moreover, controllable generation methods, techniques to integrate external commonsense knowledge and task-agnostic architectures will be studied in order to reduce the phenomena known as hallucination.
This PhD thesis is based on the hypothesis that integrating external commonsense knowledge along with controllable text generation techniques in an efficient architecture will help to reduce the hallucination issue, and besides performing similarly to state-of-the-art models. Thus, the main objective of this research is the proposal of an efficient architecture that could achieve a good performance in different NLG tasks, e.g. text summarisation, and text simplification, and could reduce hallucination as much as possible. In order to complete this main objective, several sub-objectives have been proposed:
• A1. To explore optimal controllable text generation techniques. The planned schedule of these sub-objectives can be seen in Figure 2, starting from February 2023. Group A corresponds with the study and test of state-of-the-art techniques. After an initial study, during Group B, an efficient architecture will be proposed, tested and compared with other open-source architectures using a common benchmark. Finally, in Group C the proposed architecture will be adapted to perform in different NLG tasks.
The proposed methodology to carry out this research is based on a complete and comprehensive training in all areas of NLG, including general training on NLP. After having the basic notions of NLG, the research focuses on an exhaustive analysis of the state of the art of NLG, especially on deep learning techniques that allow controlled language generation and integrate commonsense knowledge. Subsequently, the experimentation also starts, testing different open-source architectures, along with the most relevant studied techniques. After having tested several architectures, an efficient base model will be proposed, integrating commonsense knowledge and controllable generation techniques into it. Then, it will be evaluated against other architectures using a common benchmark. Finally, the proposed architecture will be adapted to perform different tasks.
At present, I am experimenting with the CommonGen dataset [34]. The CommonGen dataset consists of a set of common concepts and some reference sentences using those concepts and the main idea is to test machines for the ability of generative commonsense reasoning. I am testing with different types of approaches such as SimpleNLG, Factorised Language Models, or Neural Models over this dataset. With the proposed experiment, the main idea is to combine the best-obtained architecture with controllable generation techniques in order to obtain a base model.
In order to advance towards an effective and efficient approach for controllable text generation, several research issues are suggested and briefly discussed.
What does controllable text generation mean, and what are the most efficient methods to incorporate it? Controllable text generation is the task of producing text in a way that its attributes can be controlled [35]. These attributes can adopt a wide variety of ranges, such as stylistics, to include specific information in the content, based on the demographic attributes of the interlocutor, etc. As seen in [36], there are three ways to approach controllable text generation.
Training data in LLMs can be unbalanced due to the fact that it is difficult to balance that huge amount of data. Modifying hyperparameters may generalise the knowledge better and consequently raise obtained results.
To fine-tune a pre-trained model with more information than just the text could enhance its performance.
Using internal control variables could enrich the generation with specific capabilities.
What is hallucination and what are the ways to reduce its occurrence? Hallucination in NLG occurs when a text generated by an AI lacks of coherence or deviates from the intended sense of the source input [5]. It can be classified into two categories: intrinsic hallucinations, which appear when the generated text contradicts the source input, and extrinsic hallucinations which arise when the source input cannot substantiate the generated text.
There exist different types of approaches to minimise the occurrence of hallucinations. Firstly, constructing a reliable dataset, which does not contain any type of contradiction in the data. Secondly, modifying the encoder/decoder architecture can enhance the ability to better understand and represent the knowledge. Thirdly, proposing an optimal training strategy such as controllable text generation could benefit the model. Finally, one important approach is to integrate external commonsense knowledge into the models.
How to integrate external commonsense knowledge? Commonsense knowledge is an important factor in human communication, as it facilitates inference without the explicit mention of context [37]. Although current state-of-the-art models exhibit some common sense abilities, it is not complete yet. Traditionally, commonsense has been injected into NLG systems in the form of rules and ontologies. Nowadays, the approaches have focused on injecting commonsense into neural NLG models through pre-trained models and using commonsense graphs. But there is still much work to do in this field in order to reach a complete commonsense knowledge.
Can a smaller architecture obtain similar performance than LLMs? There are some structures such as Plug and Play models or Variational Autoencoders that are more efficient than LLMs. Integrating commonsense knowledge and controllable generation techniques into these models could help to perform like LLMs while being smaller and more efficient models.
This research work is part of the R&D projects "CORTEX: Conscious Text Generation" (PID2021-123956OB-I00), funded by MCIN/ AEI/10.13039/501100011033/ and by "ERDF A way of making Europe"