Large Language Models (LLMs), which take advantage of the Transformers architecture, have obtained remarkable outcomes in the Generative Artificial Intelligence (AI) field. Specifically, these models have boosted the Natural Language Generation (NLG) field to new dimensions. Nonetheless, state-of-the-art NLG models also entail some challenges. Firstly, these models can generate text which may contain biased or hallucinated information, that can be used in an unethical way. Secondly, these models sometimes fail at generating commonsense knowledge, a fundamental factor in the human language. Thirdly, the expense of training LLMs is excessively high. Finally, most of the NLG research proposing more eficient architectures is focused on the English language. Thus, other languages such as Spanish still lack resources and models to address the NLG. Given the challenges mentioned above, the main objective of this paper is to provide a detailed research line that aims to propose an eficient and efective text generation architecture that could generate high-quality text in Spanish.
Generative Artificial Intelligence (AI) is a rapidly growing trend involving machine learning algorithms
to construct systems capable of generating new content. This trend has its origin in neural networks,
which were one of the earliest forms of generative AI [
One of the important topics within the generative AI trend is the Natural Language Generation
(NLG) field. NLG is a sub-field of Natural Language Processing (NLP) that aims to generate natural
language to achieve a specific communicative goal [
Large Language Models (LLMs) are the core of those NLG tools. These models, formed by millions
of parameters in their neural networks are extremely expensive to train. Therefore, there is a need
to find more eficient architectures to generate text. Moreover, LLMs and NLG are not exempt from
mistakes and one of the current major issues is the hallucination phenomenon, which is when a text is
nonsensical or unfaithful to the provided source [
Another important issue is bias, which is the misrepresentation errors that favour certain groups [
Finally, these tools also lack logical reasoning or commonsense, a vital factor in human intelligence
[
These limitations can be exploited in a bad and unethical manner to generate misinformation.
Furthermore, there is a notable disparity in NLG research. The majority of studies conducted are on
the English language [
Because of this, our goal with this research proposal is to describe my thesis project which focuses on creating an eficient architecture that integrates external commonsense knowledge along with controllability techniques to enhance the performance of a smaller model and solve the hallucination problem.
This section aims to contextualise this research project within the NLG state of the art.
NLG is the sub-field in the Natural Language Processing (NLP) area that aims to produce meaningful
sentences to meet a communicative goal [
In order to address those tasks, NLG systems have evolved through diferent architectures. Originally,
the NLG task consisted of a sequential scheme of three diferent stages (macroplanning, microplanning
and realisation). Macroplanning is the set of sub-tasks related to the selection of what information
to include in the generated text. Microplanning receives as input the output of the Macroplanning
stage and conducts diferent sub-tasks to decide how to include that selected information in the final
text. Finally, the realisation stage receives the generated plan from the previous steps and performs the
generation syntactically correctly. Modular architectures is the group of architectures that follows
this scheme. They make a well-diferentiated distinction between the distinct sub-tasks of each stage.
Reiter proposed the architecture that was considered the standard within this group [
As the NLG field started to become mature, the distinction between sub-tasks became more flexible, performing the generation in fewer steps. This group of architectures is named planning perspectives. This scheme was similar to the modular architecture but needed fewer sub-tasks in each stage.
With the appearance of neural networks, the sub-task division disappeared, originating global
approaches. This group performs the generation in one step. The most important milestone in this
group was the proposal of Transformers architecture [
Commonsense knowledge is an important factor in human communication, as it facilitates inference
without the explicit mention of context [
Originally, commonsense has been incorporated into NLG systems through rules and ontologies.
Since the neural networks proposal, the focus has shifted to integrating commonsense into neural NLG
models through pre-trained models and commonsense graphs. However, there is still much work to
be done in this field to achieve complete commonsense reasoning and generation. Although current
state-of-the-art models exhibit some commonsense abilities, it is far from perfect. The commonsense
knowledge integration into human language is a challenging task in the NLG field [
As in any other NLP task, NLG systems also should be evaluated. The generated output usually is compared against a reference or reference text through diferent evaluation metrics. Commonly the NLG evaluation metrics can be divided into three categories: • Human evaluation metrics are the best and more reliable metrics [20]. Nonetheless, they also present some problems: they are expensive and time-consuming because they require a great deal of manual work, and replicability is hard to achieve due to the idiosyncrasies of every evaluator. • Standard evaluation metrics are based on string or n-gram techniques to measure the distribution similarity via overlaps with a reference text. These metrics are not fully aligned with human judgements, but even so, they are temporary and computationally low-cost. Most of these metrics evaluate the quality of a text by comparing it against a reference text based on features such as words, characters or embeddings. Metrics measuring the word overlapping between a candidate sentence and a reference sentence are the most common. Some examples are: BLEU [21], ROUGE [22], CIDEr [23], or SPICE [24]. Character overlapping performs better in languages with lexical diversity. Some metrics within this group are Extended Edit Distance [25] or chrF [26]. Finally, embeddings-based metrics tend to capture better semantic similarity. BERTscore [27], and Word Mover-Distance [28] are examples of this group. • Machine-learned evaluation metrics can generate a more reliable score given that during training, models learn how to generate and evaluate text in a more human-like way. BARTScore [29] and GPTScore [30] are metrics that use BART or GPT models to evaluate the generated text.
This PhD thesis hypothesises that integrating external commonsense knowledge into a smaller architecture in terms of parameters, compared to state-of-the-art LLMs, will boost the quality of the outputs when generating text. Those texts could be similar to the ones that could write a human when solving certain tasks. Moreover, external commonsense knowledge will help reduce the problem of hallucinations. After reviewing the scientific literature, we have observed that some works have been conducted to incorporate external commonsense in NLG systems. However, although they improved the results of previous works, the performance was still not excellent. Moreover, most of the research has been conducted in the English language. Therefore, the main objective of this research is to design an eficient architecture that will require less computational expenses than the state-of-the-art LLMs. This architecture will be capable of achieve human-like performance in generating text for diferent NLG tasks prioritising the research for Spanish. Additionally, a key focus will be on minimising the issue of hallucinations.
To fulfil the objectives of this PhD thesis, we have proposed a methodology consisting of four main important milestones, that will be explained next: define a NLG pipeline; analyse, test and propose NLG architectures; analyse and build a corpus in Spanish; and evaluate proposed NLG architectures. 1. Define a NLG pipeline. The first step of my research was to propose a pipeline to follow. It involves the steps of defining the task I want to address, the corpus and external knowledge collection, how to integrate both in a NLG architecture and finally evaluating the results obtained. Figure 2 shows the sequential steps of this pipeline. The task this research project is addressing ifrst is the commonsense generation through a concept-to-text task. Given some words, the goal is to generate a sentence that contains those words and has common sense. Afterwards, more NLG tasks will be addressed. 2. Analyse, Test and Propose NLG Architectures. During the initial part of the research, we explored diferent available NLG architectures in English and Spanish to find a suitable model to address the NLG task based on the topics pursued. Nowadays, most of the architectures employed are based on LLMs. Despite this, we will first test and compare traditional and neural architectures to efectively decide which architecture performs better based on a set of decisions, such as content produced, eficiency, etc. Consequently, we will try to integrate commonsense knowledge and controllable generation techniques into the best-performing architecture obtained from our experimentation to raise its performance and compare the results against the state-of-the-art models. 3. Analyse and Build Corpus in Spanish. Datasets constructed for a specific task in Spanish are hard to find, as most of the available corpora are in the English language. Moreover, in some cases, these datasets may be biased [31]. Once we have proposed an eficient architecture, we will focus on adapting that architecture to address diferent NLG tasks in Spanish, such as commonsense generation, text simplification, or abstractive summarisation. Therefore, we will have to collect and build specific corpora in Spanish to address these tasks. Moreover, as we are concerned about the importance of the training data on the results of the NLG model, we will check and preprocess, if necessary that corpora to try to be unbiased and balanced. 4. Evaluate Proposed NLG Architecture on diferent tasks. An important part of this research is to measure the performance of our proposed model. Most NLG metrics tend to compare the generated text against some reference texts. As NLG systems can generate texts in a large number of diferent styles, these metrics may not be appropriate. To be able to determine which metric is the most appropriate to measure the performance of each proposed task an exploratory analysis will be conducted and compared with a manual evaluation of those tasks.
Two experiments have been conducted: a preliminary experiment on English NLG architectures and the problems in evaluating its results, and a similar experiment in Spanish.
1. English NLG models evaluation: This experiment is detailed in [32]. We conducted an experiment
for the task of commonsense generation based on the CommonGen dataset [
We also performed a manual and automatic evaluation. The manual evaluation shows that the produced sentences are not acceptable. Furthermore, in some cases, the generated sentences do not contain all the words given as keywords. For the words "despedir", "aeropuerto", and "amigo" the system generated the sentence "él se despidieron que vinieras al aeropuerto.". We also evaluated automatically the sentences with the same metrics as for the English experiment. Table 1 shows the results obtained by this model in diferent NLG evaluation metrics and compared with the results obtained for the T5 model in English of the previous experiment. Although they are two diferent datasets, we can see that the results achieved for Spanish are lower than the results obtained in the English experimentation. Those results align with the manual evaluation, which showed a worse performance for the Spanish model than for the English model.
Model T5-Small Spanish T5-Small English
SPICE
Considering the obtained results, the next steps are the experimentation with diferent architectures that integrate external commonsense and the search for the optimal way to evaluate the generated outputs, which can be done manually or to find other more appropriate automatic metrics. 1. External commonsense Integration: To figure how to obtain external knowledge for the task of concept-to-text and how to include it in the proposed model are two key points to incorporate commonsense in NLG architectures. There are diferent options such as ConceptNet, WordNet or COMET that could be interesting to explore. The research question is how to successfully extract knowledge from them and incorporate it into a NLG system. 2. Output evaluation: As demonstrated in [32], most common automatic evaluation metrics do not align completely with human performance, only measuring the overlapping with a reference sentence. Thus, finding other methods to evaluate the generated text is important to measure the performance of the models. Some works already done to measure hallucinations could be interesting to explore to evaluate outputs. Examples are AlignScore metric [34] and Hallucination_evaluation_model [35].
As I explained, my research is currently focused on the task of concept-to-text generation. This task is based on giving some concepts, the systems must generate a sentence using those concepts. We have compiled a Spanish corpus named COCOTEROS to accomplish this task. The next step is to incorporate external commonsense into some pre-trained models to enhance the results obtained. Given that, the research questions that arise are the following: • Which is the most adequate external knowledge base to complement this task? • In which part of the pipeline is more eficient to incrust the knowledge, during fine-tuning, during inference, or in both? • How the hallucination issue can be measured for this task in Spanish? • If we use LLMs to address this task, will the results be improved considerably?
This research work is part of the R&D projects “CORTEX: Conscious Text Generation” (PID2021123956OB-I00), funded by MCIN/ AEI/10.13039/501100011033/ and by “ERDF A way of making Europe” text generation challenge for generative commonsense reasoning, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 1823–1840. URL: https://www.aclweb.org/anthology/2020.findings-emnlp.165. [19] F. Carlsson, J. Öhman, F. Liu, S. Verlinden, J. Nivre, M. Sahlgren, Fine-grained controllable text generation using non-residual prompting, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 6837–6857. [20] K. R. Chandu, A. W. Black, Positioning yourself in the maze of neural text generation: A task-agnostic survey, 2020. URL: https://arxiv.org/abs/2010.07279. doi:10.48550/ARXIV.2010. 07279. [21] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318. [22] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004, pp. 74–81. [23] R. Vedantam, C. Lawrence Zitnick, D. Parikh, CIDEr: Consensus-based image description evaluation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 4566–4575. [24] P. Anderson, B. Fernando, M. Johnson, S. Gould, SPICE: Semantic propositional image caption evaluation, in: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, Springer, 2016, pp. 382–398. [25] P. Stanchev, W. Wang, H. Ney, EED: Extended edit distance measure for machine translation, in: Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), 2019, pp. 514–520. [26] M. Popović, chrF: character n-gram f-score for automatic MT evaluation, in: Proceedings of the tenth workshop on statistical machine translation, 2015, pp. 392–395. [27] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, BERTScore: Evaluating text generation with bert, arXiv preprint arXiv:1904.09675 (2019). [28] M. Kusner, Y. Sun, N. Kolkin, K. Weinberger, From word embeddings to document distances, in:
International conference on machine learning, PMLR, 2015, pp. 957–966. [29] W. Yuan, G. Neubig, P. Liu, BARTScore: Evaluating generated text as text generation, Advances in Neural Information Processing Systems 34 (2021) 27263–27277. [30] J. Fu, S.-K. Ng, Z. Jiang, P. Liu, GPTScore: Evaluate as you desire, arXiv preprint arXiv:2302.04166 (2023). [31] J. S. Ernst, S. Marton, J. Brinkmann, E. Vellasques, D. Foucard, M. Kraemer, M. Lambert, Bias mitigation for large language models using adversarial learning (2023). [32] I. Martínez-Murillo, P. Moreda, E. Lloret, Analysing the problem of automatic evaluation of language generation systems, Procesamiento del Lenguaje Natural 72 (2024) 123–136. [33] M. M. Maestre, I. Martínez-Murillo, E. Lloret, P. Moreda, A. Suárez Cueto, COCOTEROS: A spanish corpus with contextual knowledge for natural language generation, in: 40th Annual Conference of the Spanish Association for Natural Language Processing 2024: Posters (SEPLN-P 2024), CEUR, 2024. [34] Y. Zha, Y. Yang, R. Li, Z. Hu, AlignScore: Evaluating factual consistency with a unified alignment function, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 11328–11348. URL: https://aclanthology. org/2023.acl-long.634. doi:10.18653/v1/2023.acl-long.634. [35] S. M. Hughes, M. Bae, Hughes hallucination evaluation model (hhem) leaderboard, Huggingface, available at https://huggingface. co/spaces/vectara/leaderboard (accessed 22nd April, 2024) (2024).