The “Conscious Natural Text Generation” project (CORTEX, grant ref. PID2021-123956OB-I00) investigates how Natural Language Generation (NLG) architectures can be improved so that they can exploit external and commonsense knowledge, and integrate it in a controllable manner (i.e., determining what knowledge to include and how to include it) when generating new text. Our main scientific objective is, therefore, to investigate and propose novel NLG architectures that integrate diferent types of knowledge to automatically produce reliable, truthful and quality texts, whose information complies with the principles of objectivity and plurality, thereby minimizing bias or manipulated content. The new generation of NLG systems obtained from CORTEX will significantly improve the semantic quality of the generated text, preventing, among other phenomena, the inclusion of invented facts that do not match reality (i.e., hallucination).
the text that is being generated. Thus, there are no guarantees that the information generated automatically is Natural Language Generation (NLG) oversees the pro- accurate and factually correct; it could be distorted in duction of text or speech in a coherent and appropri- some way. This could lead to the phenomenon known ate way for the desired communication objective. NLG as hallucination [1], a term used to describe the generaencompasses many types of applications — e.g. sum- tion of fictitious information that has nothing to do with marisation, report generation, chatbots— pursuing very reality. The potentially dangerous consequences of this diferent communicative goals and allowing for a wide are erroneous interpretations, the spreading of false invariety of inputs and outputs. formation, manipulation, bias that may result from the
Significant progress has been made recently in NLG exclusion of important facts which are not aligned with due to the advent of deep neural models and end-to-end a stance, etc. architectures and their capacity to produce fluent ex- The “Conscious Natural Text Generation” project, cerpts of meaningful text together with their capability whose acronym is CORTEX, is funded by the Spanish to capture longer dependencies within the data or even Government with grant reference “PID2021-123956OBcontext information, as the architectures based on Trans- I00” and developed by the GPLSI research group of the formers (GPT-3 or T5). However, despite their achieve- University of Alicante. CORTEX is dedicated to researchments, these approaches face several challenges in that ing the improvement of NLG architectures by integrating they require huge volumes of data, there is a lack of trans- and injecting external commonsense knowledge. It is parency, and constraining or controlling the algorithms expected that the new generation of NLG systems develinvolved is dificult. In fact, among the limitations of oped by CORTEX will have a degree of consciousness, or these new language models there is one that concerns at least a degree of common sense, thus making a signifithe unintended consequences related to the generation cant contribution to improving the semantic quality of of distorted text. This can happen when the language the generated text, avoiding, among other phenomena, model is incapable of understanding or being aware of the inclusion of invented facts that are not in accordance with reality. In fact, it is said that “common sense reasoning is the new frontier of Artificial Intelligence (AI)".
The initial hypothesis of CORTEX is that the efective integration of world and external knowledge in NLG architectures improves the commonsense reasoning capabilities of NLG systems, which is needed to automatically produce accurate, correct, and reliable texts that will be in line with real facts. Therefore, the research questions that must be asked to validate our hypothesis are the following: 1. What generic language models exist? 2. What information and characteristics must knowledge sources have to enable a NLG system to generate syntactically and semantically correct texts that are also adequate according to the required communicative situation? 3. How efectively would the proposed NLG approaches be able to integrate, use and adapt large generic language models and commonsense knowledge?
A gender-balanced (4 women and 5 men) multidisciplinary research team of 7 doctors and 2 PhD students, has been formed. All of them belong to the GPLSI research group and they have Computer Science or Linguistics background.
The methodology has been organised into three main Work Packages (WP) whose tasks and results are interlinked, as shown in Figure 1. An additional transversal fourth WP concerning the management and dissemination of information and resources is also crucial for the efective implementation of the project. Next, each of the WPs and its tasks is explained in more detail.
and Infrastructures for Natural Language Genera
The answer to the previous research question embod- tion ies the main scientific objective of the project, which can The purpose of this work package is to explore multibe defined as: “To investigate and propose knowledge- ple and heterogeneous knowledge sources and existing enhanced NLG architectures which integrate diferent infrastructures. Semantic world knowledge is essential kinds of knowledge to automatically produce reliable, for resolving many deep and complex decisions in natutruthful, and quality texts whose information complies ral language understanding and generation. Moreover, with principles of objectivity and plurality, avoiding bias to eficiently and efectively manage all existing and poor manipulated content”. This main objective is divided tential new knowledge, an appropriate infrastructure is into the following specific objectives: also necessary. Given its importance, objectives OB1 and • OB1. Gather and analyse the available generic OB2 will be achieved through the successful completion existing language models and knowledge sources of the following three tasks.
(structured and unstructured). Task 1.1. Exploration of existing knowledge • OB2. Determine what kind of knowledge is most sources and language infrastructures: The objecappropriate to improve the NLG process, as well tive of this task is to explore and analyse in-depth so as compile, and adapt models that will enable the as to compile existing and available knowledge, infrassemantic enrichment of NLG approaches. tructures and language models, thereby identifying the potential of these resources as well as their limitations • OB3. Research, propose and develop novel NLG for multilingual NLG. This task will produce a specific approaches that can integrate, be guided by, or computationally appropriate knowledge compilation for simply use the obtained knowledge, thus leading NLG. Moreover, it explores to what extent the available to more accurate, flexible, and dynamic common large language models and infrastructures can be used sense and conscious generation approaches. as a basis for further research in tasks 1.2, 1.3, as well as • OB4. Propose and develop various scenarios and the forthcoming work packages.
use cases that demonstrate the validity and appli- Task 1.2. Knowledge quality assurance and excation of the NLG task. traction: Ensuring high knowledge quality and precision • OB5. Evaluate intrinsically and extrinsically each is crucial to create NLG models that learn to avoid incorof the proposed techniques and approaches and porating societal biases and inaccurate information into scenarios with the most suitable standard metrics, further steps [2]. Hence, the aim of this task is to define a or create novel metrics, if necessary. methodology and metrics to analyse and detect possible • OB6. Promote and disseminate the results ob- biases in the knowledge sources analysed in Task 1.1 and tained from the project through diferent national to extract relevant patterns which may be also used on and international media, as well as exploit the po- succeeding tasks. tential for transferring this technology to society. Task 1.3. Knowledge discovery and representation: The goal of this task is to centralise and represent 3. Team and Work Plan the obtained knowledge that contains heterogeneous and multilingual information. This will be initially The research work addressed in the CORTEX project done by exploring available corpora and knowledge will have a duration of three years, starting from 1st graphs that are used for commonsense reasoning, such September 2022. as CommonsenseQA [3], ConceptNet [4] or AGENDA [5]. which point pragmatic features of language, such as com
WP2: Commonsense knowledge-enhanced Natural municative intentions, determine the linguistic elements Language Generation that the generated text should include. This will enable
The main goal of this WP is to analyse and propose a narrowing down of the generation process to produce novel and cost-efective NLG approaches that integrate content that is conscious of its pragmatic context, going commonsense knowledge acquired from WP1 within the beyond the lexical, syntactic and semantic features used generation process, so that a new generation of com- so far in the state of the art. mon sense and conscious NLG systems can be produced. Task 2.2. Analysis and comparison of Natural Several tasks related to NLG architectures and how to Language Generation architecture types: The goal integrate knowledge has been proposed in order to ac- of this task is to find a flexible but efective and eficomplish objectives OB3 and OB5. cient NLG architecture. The architecture of a NLG ap
Task 2.1. Definition and adaptation of language proach determines how the aforementioned sub-tasks representation models for Natural Language Gen- (macroplanning, microplanning and surface realisation) eration: Based on the existing language infrastructure are integrated in the generation process. This task will analysis conducted in WP1, the objective here is to de- explore and experiment with diferent types of architectermine: tures. These include: i) sequential architectures, where 1. Wfcnoeirnhssrgice,[hp6mr]meicasoerndondeptlilisnatsgnarnsheuiunmbmg-otaaarnenskldaaspns,puggrureofanapgcereeriaafrtoleelryaatlhnimsedaaNtecifeorLocnGtpi[pvl7are]no.-- ittushhleaejostd;ibniiifeet)lrnyienenpfittetefgrrsorfuoambrt-emttdahesaedkrcasahdtaivortaeennccutteuna;rgdaeeenssr,dtwoaifikhib)eeonhretyihtnbhroseifdewtpahahrercomahleti.etpemrcotoucderes-ss
Task 2.3. Proposal and development of knowl2. How they can be adapted to obtain specific lan- edge integration approaches in Natural Language guage models depending on the targeted NLG Approaches architectures: The purpose of this task is tasks to be addressed. to analyse how to incorporate commonsense knowledge into downstream NLG models. For this purpose, diferent options can be explored [8], including the following: Additionally, some linguistic information such as the communicative intention of the message to be created will also be considered for the comparison of the diferent models that could be employed in the NLG tasks. This feature would enable the automatic change in form of the generated text depending on the intention to be accomplished. Consequently, this task will also analyse up to pretraining on commonsense knowledge bases or explanations; 3. Using multi-task objectives with common sense
relation prediction.
In this task, we first plan to explore the use of knowledge already available in semantic networks, but also including new knowledge obtained from WP1 through the existing corpora. In parallel, our aim is also to determine to what extent neural language models (e.g. Transformers) can be modified and fine-tuned to integrate common sense knowledge during the NLG process.
Task 2.4 Natural Language Generation Evaluation: The purpose of this task is to evaluate every intermediate or final result associated with the previous tasks. These results can be evaluated from diferent perspectives depending on the goal of the evaluation [9]. Within this context, we can use extrinsic methods (to determine whether the application achieves its objective) and intrinsic methods (to examine the system’s performance and the quality of its output). Moreover, existing and new challenges that fit within our scope will also be used as a means of evaluating and comparing our NLG approaches with respect to other methods developed by the research community.
This last WP will contribute to fulfilling OB4 and OB5, and its purpose is to apply the proposed and developed knowledge-enhanced NLG approaches into diverse scenarios and use cases to validate and show their appropriateness in real contexts. Each scenario will integrate the findings and outcomes of WP1 and WP2, and they will be evaluated with the specific and standard metrics appropriate for the diverse settings. In particular, the three following scenarios are envisaged.
Task 3.1 Text Summarisation: Text summarisation aims to synthesise information keeping only what is relevant [10]. Although research into extractive approaches is the most predominant, they are limited to literally copying the information from the input and pasting it in the output summary. On the other hand, abstractive summarisation is more powerful, but at the same time more challenging. The goal of this task is to address abstractive summarisation, integrating a strong NLG component from the results of WP2. The integration of a knowledgeenhanced NLG component during the abstractive summarisation process would contribute to producing more human-like summaries, as it will be possible to detect and infer relevant information, even when this is described through complex events in several non-consecutive sentences.
Task 3.2 Creative text generation: One of the most complex NLG scenarios is the production of creative or artistic texts, including storytelling or poetry [11]. In both cases, a NLG system must deal with specific linguistic phenomena such as the type and structure of the narrative events, prosodic devices such as meter and rhythm, among others., or temporal or causal relations between events. During this sub-task, we will analyse and automatically extract literary events, their structures and the temporal or causal relations between them [12, 13]. We also will explore formal analysis of metre and rhythm in a corpus of poetry to introduce more realistic prosody in NLG.
Task 3.3 Chatbots for emotional intelligence: Emotional Intelligence education [14] is a pending issue for society that could potentially contribute to solving many current social problems. Chatbots can be beneficial for helping users to improve their emotional intelligence and to better manage and understand their emotions. Specifically, the chatbot will work on “tales with a message”. These folk stories are appropriate in that they represent the millennia-long tradition of homo sapiens to skilfully transmit and understand knowledge. We will apply the research conducted to the text generation techniques of the previous WPs to help users develop social cognition (the ability to identify and understand social situations [15]), as well as to improve their reading comprehension level.
Task 3.4 Making English-language metaphors more intelligible: The study of metaphors in specific domains in the English language is motivated by the desire to promote inclusivity and falls within the area known as English for specific purposes. Indeed, there is a pressing need to process metaphors in a common language for all communities as they are often ambiguous and require up to date global knowledge to understand their meaning and purpose [16]. Thus, the goal of this task is to facilitate the human assimilation of metaphors and to provide an equivalent meaning in a simpler manner.
The goal of this transversal WP is to oversee the internal communication flow throughout the duration of the project to ensure that the objectives are met via the intermediate and final results of each task. This WP will also promote the correct dissemination of the project thereby achieving objective OB6. The main instrument for communicating relevant results will be publications in the most relevant conferences and journals related with the project.
Concerning the scientific and technical impact, our project focuses on theoretical cutting-edge research with many applied ofshoots for NLG. This will lead to several [4] R. Speer, J. Chin, C. Havasi, Conceptnet 5.5: An applications designed for facts/data summarisation, sto- open multilingual graph of general knowledge, in: rytelling and narrative generation, chatbots for specific Proceedings of the Thirty-First AAAI Conference purposes, and facilitating the comprehension of complex on Artificial Intelligence, AAAI’17, AAAI Press, expressions in specific domains. Tools that achieve these 2017, p. 4444–4451. applications are expected to have a great demand in the [5] R. Koncel-Kedziorski, D. Bekal, Y. Luan, M. Lapata, next decade. H. Hajishirzi, Text Generation from Knowledge
As diferent types of architectures will be explored for Graphs with Graph Transformers, in: ProceedNLG during the execution of the CORTEX project, we ings of the 2019 Conference of the North American will be able to determine which ones maximise the results Chapter of the Association for Computational Linwith minimum computational cost, thereby providing a guistics: Human Language Technologies, Volume set of cost-efective NLG methods. 1 (Long and Short Papers), Association for Com
Given the feasibility of incorporating these models putational Linguistics, Minneapolis, Minnesota, and approaches into prototypes or demos, there will be 2019, pp. 2284–2293. URL: https://aclanthology.org/ high social and commercial applicability of the research N19-1238. doi:10.18653/v1/N19-1238. through the proposed scenarios, enabling the promo- [6] C. Dong, Y. Li, H. Gong, M. Chen, J. Li, Y. Shen, tion of the project’s many potential applications to re- M. Yang, A survey of natural language generation, gional, national and international companies and other ACM Comput. Surv. 55 (2022). URL: https://doi.org/ stakeholders. Knowledge discovery and integration from 10.1145/3554727. doi:10.1145/3554727. multimodal, multilingual and heterogeneous information [7] A. Gatt, E. Krahmer, Survey of the state of the art sources, in turn, opens up many unexplored theoretical in natural language generation: Core tasks, appliand applied research avenues for the NLG international cations and evaluation, J. Artif. Int. Res. 61 (2018) community, which are expected to attract interest from 65–170. the Natural Language Processing and Artificial Intelli- [8] W. Yu, C. Zhu, Z. Li, Z. Hu, Q. Wang, H. Ji, M. Jiang, gence research communities. A survey of knowledge-enhanced text generation, ACM Comput. Surv. 54 (2022). URL: https://doi.org/ 10.1145/3512467. doi:10.1145/3512467.
Acknowledgements [9] A. Celikyilmaz, E. Clark, J. Gao, Evaluation of text generation: A survey, arXiv preprint This research work is part of the R&D project CORTEX: arXiv:2006.14799 (2020).
Conscious Natural Text Generation (PID2021-123956OB- [10] A. A. Syed, F. L. Gaol, T. Matsuo, A survey of the I00), funded by MCIN/ AEI/10.13039/501100011033/ and state-of-the-art models in neural abstractive text by “ERDF A way of making Europe”. summarization, IEEE Access 9 (2021) 13248–13265. [11] A. I. Alhussain, A. M. Azmi, Automatic story References generation: A survey of approaches, ACM Comput. Surv. 54 (2021). URL: https://doi.org/10.1145/ [1] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, 3453156. doi:10.1145/3453156.
Y. J. Bang, A. Madotto, P. Fung, Survey of hal- [12] M. Sims, J. H. Park, D. Bamman, Literary event delucination in natural language generation, ACM tection, in: Proceedings of the 57th Annual Meeting Comput. Surv. 55 (2023). URL: https://doi.org/10. of the Association for Computational Linguistics, 1145/3571730. doi:10.1145/3571730. 2019, pp. 3623–3634. [2] E. Sheng, K.-W. Chang, P. Natarajan, N. Peng, So- [13] A. Feder, N. Oved, U. Shalit, R. Reichart, Causalm: cietal biases in language generation: Progress and Causal model explanation through counterfactual challenges, arXiv preprint arXiv:2105.04054 (2021). language models, Computational Linguistics 47 [3] A. Talmor, J. Herzig, N. Lourie, J. Berant, Com- (2021) 333–386.
monsenseQA: A question answering challenge tar- [14] D. Goleman, Emotional Intelligence, Bantam Books, geting commonsense knowledge, in: Proceedings New York, 1995. of the 2019 Conference of the North American [15] J. Uekermann, M. Kraemer, M. Abdel-Hamid, B. G. Chapter of the Association for Computational Lin- Schimmelmann, J. Hebebrand, I. Daum, J. Wiltfang, guistics: Human Language Technologies, Volume B. Kis, Social cognition in attention-deficit hyper1 (Long and Short Papers), Association for Com- activity disorder (adhd), Neuroscience & biobehavputational Linguistics, Minneapolis, Minnesota, ioral reviews 34 (2010) 734–743. 2019, pp. 4149–4158. URL: https://aclanthology.org/ [16] S. Rai, S. Chakraverty, A survey on computational N19-1421. doi:10.18653/v1/N19-1421. metaphor processing, ACM Computing Surveys (CSUR) 53 (2020) 1–37.