The exponential growth of available data to Digital Humanities (DH) has created an impending need for tools capable of analyzing and extracting information from multi-lingual historical documents. This paper explores the research directions of my PhD project: providing DH scholars with efective, eficient, and explainable tools based on recent advancements in Large Language Models (LLMs). Two are the main directions of investigation: Self-Improving LLMs applied to Text-to-SQL and Topic Modeling, with a focus on interacting with and augmenting existing DBMS; Knowledge Graph (KG) creation and integration to mitigate hallucination, improve transparency and reasoning in question-answering systems. At the heart of my research lies the Digital Maktaba (DM) project which seeks to create a digital library for assisting in the preservation and analysis of multicultural non-latin heritage documents using, among others, cutting edge techniques for Natural Language Processing (NLP) and Data Science. The DM objectives and ideals align with the ultimate goal of the PhD project: the creation of instruments capable of aiding human-data interaction and information extraction while keeping the user at the center of an ever-evolving system. These tools have the potential to revolutionize the way DH scholars interact with historical documents, leading to new insights and discoveries for the field at large.
Digital Humanities (DH) is an interdisciplinary field that merges humanities research with digital technologies with the goal of revolutionizing the analysis of cultural and historical artifacts. The exponential growth of available digital data has led to new challenges in accessing, studying and comparing this vast amounts of information, creating the need for more advanced tools that may assist the user in the analysis and cataloging of documents, be it for the major latin languages or non-latin ones.
To assist in this endeavor new algorithms based on recent developments in Natural Language Processing (NLP) and, more specifically, Large Language Models (LLMs) are gaining traction. These methods harness advancements in the former to enable cross-lingual data mining and language processing, facilitating extraction of information and user interaction with vast databases without the need for extensive technical expertise or costly human intervention.
In this research project, my primary focus will be on leveraging newly developed methodologies for self-improving LLMs to dynamically adapt to underlying data distributions and improve performances in a self-supervised manner. This will be initially used for improving current techniques of Text-to-SQL and Topic Modeling for the interaction and augmentation of DBMS.
Another area of research will be on transparency, hallucinations and reasoning. By Knowledge Graph (KG) creation, augmentation and integration, I seek to provide, on one side, easily accessible and human readable explanations of the model knowledge and planning steps and, on the other, explicit and reusable thought processes.
In this paper, I first provide, in Section 2, an overview of the Digital Maktaba case study, pointing out its objectives and significance within, and not limited to, the landscape of digital humanities. Following that, in Section 3, I will discuss the related works that will act as the foundation for my research. In Section 4 I will provide a comprehensive overview of my ideas, addressing current limitations and outlining strategies for improvement. Finally, in the concluding section, a summary of all the points explored in this paper will be given.
In this PhD, the Digital Maktaba (DM), defined as WP5 in the ITSERR project, functions as the primary source and ultimate goal. This work package is dedicated to crafting a digital library that can analyze and extract information from multi-lingual documents, particularly from Arabic scripts (Arabic, Persian and Azerbaijani).
To facilitate this endeavor, multiple books have been provided from the “Giorgio La Pira” library in Palermo, hub of the FSCIRE foundation dedicated to history and doctrines of Islam. This repository of high-quality documents given to the DM project will be invaluable for testing new ideas and techniques, particularly in languages with limited available resources.
My contributions toward realizing the DM’s overarching objectives involve providing tools that facilitate the exploration of vast datasets, extraction of pertinent information, and answering complex questions. In Figure 1, we see a representation of the main modules that I identified in this early stage of research.
• Text-to-SQL: Retrieval-Augmented Generation (RAG) methods have been shown to
reduce hallucinations and improve the quality of responses [
The capability of Large Language Models to enhance their own performance through
selftraining and self-supervision represents a significant advancement in the field. Early
methodologies aimed at improving model performance on specific tasks such as code generation [
Another branch of research, instead, focused on using highly performant and expensive
models to generate high-quality data to train smaller ones [
However, the cutting edge of the domain is moving towards self-improving methodologies
for creating better-performing models at a low human cost. In [
Concurrently, [
The realm of Text-to-SQL translation has seen considerable advancements, propelled by both
traditional and innovative methodologies. Early eforts in this area often centered around templates
and heavy human engineering [
Traditional Topic Modeling approaches like Latent Dirichlet Allocation (LDA) [
In recent years, instead, attention has shifted towards utilizing autoregressive LLMs. In [14] and [15] both closed-source and open-weight models were tested for defining and assigning topics in diferent datasets and contexts, demonstrating significant improvements compared to baselines, with mixed results for open-weight models compared to closed ones.
As stated in [16] the integration of Knowledge Graphs and Large Language Models has emerged as a compelling area of research due to their complementary strengths.
LLMs excel in language understanding, generation and processing of extensive textual data. Yet, challenges in interpretability and controllability remain. Conversely, KGs provide clear, structured factual knowledge, enhancing explainability and reasoning.
Therefore synergistic methodology are an intriguing avenue of research. KGs can combat LLMs’ propensity to hallucinate facts by injecting real-world knowledge during training [17, 18] or inference [19]. In turn, LLMs can facilitate the construction [20] and enrichment [21] of KGs through their ability to process and interpret vast amounts of unstructured text. Finally, LLMs can act as a powerful tool for interacting with available knowledge bases, enabling users to ask natural language questions to receive structured and informative responses that leverage the depth and interconnectedness of KGs [22].
As described in Section 3, the current best performing techniques for both Topic Modeling and Text-to-SQL are based on closed source models but this is a more broad phenomena as shown in the current ranking for chatbots [23]. This reliance poses several problems for real-world application. These include cost estimation dificulties, dependency on third-party availability, privacy concerns, potential biases and poor transparency. These issues are particularly pronounced in digital humanities research, where the analysis of lengthy historical and multifaceted documents is central.
The objective of this PhD is to address these challenges by improving existing open-source models, thereby providing scholars with efective, eficient and explainable tools for analyzing multilingual documents and interacting with complex data systems. This will be achieved through three primary avenues: self-improvement of Text-to-SQL and Topic Modeling, and KGs integration.
In the domain of Text-to-SQL, current methodologies mainly focus on massive and closed LLMs
solutions that are a dificult proposition for a lot of real-world use cases [
The subject of research will be the use of self-improving mechanisms to train the model
in a self-supervised manner by generating new samples of natural language queries through
in-context learning as in [
Similarly, for Topic Modeling, new documents on the same subjects will be created, the associated topics will be predicted and judged using additional information that can be extracted from more traditional methodologies. This pipeline will provide interesting data for the iterative reinforcement learning procedure. The efectiveness of this specific approach, while no intermediate results are available as of the time of writing, shows promise as it has been shown that Large Language Models are good judges of topic categorization and have a strong correlation with human judgments [24].
The synergy with knowledge graphs has become more and more apparent in recent years leading us to explore their usage for LLMs and with LLMs. This research will focus on leveraging KGs to ground model responses, mitigate hallucinations, and identify conflicting information among documents. In particular, following the work of [25], I plan to use the knowledge retrieved at inference time by RAG systems to generate and augment the available knowledge graphs by self-reflecting on the contextual and reasoning information. By utilizing in conjunction the aforementioned KG alteration with their exploration, as in [22], I foresee explicit and transparent reasoning steps being saved and reused by the LLM for solving progressively more complex tasks. These capabilities will be crucial for digital humanities research, allowing deeper, richer and more transparent analysis of multi-document and multilingual subjects.
This paper presented the research objective and foundational works of my studies which aims to create instruments capable of aiding data interaction and information extraction utilizing Self-Improvement methodologies and Knowledge Graphs. The DM project has been shown as an ideal starting point for building a digital library capable of providing scholars with eficient and efective tools for the analysis and cataloguing of multi-lingual, non-latin and lengthy historical documents. Key areas of research were outlined with great emphasis placed on the application of cutting-edge techniques. This project will hopefully contribute a key piece to the puzzle in creating ever-evolving systems built with humans and for humans.
This work was supported by the PNRR project Italian Strengthening of Esfri RI Resilience (ITSERR) funded by the European Union – NextGenerationEU (CUP:B53C22001770006). [14] C. M. Pham, A. Hoyle, S. Sun, M. Iyyer, Topicgpt: A prompt-based topic modeling framework, arXiv preprint arXiv:2311.01449 (2023). [15] H. Wang, N. Prakash, N. K. Hoang, M. S. Hee, U. Naseem, R. K.-W. Lee, Prompting large language models for topic modeling, in: 2023 IEEE International Conference on Big Data (BigData), IEEE, 2023, pp. 1236–1241. [16] S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, X. Wu, Unifying large language models and knowledge graphs: A roadmap, IEEE Transactions on Knowledge and Data Engineering (2024). [17] Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, Q. Liu, ERNIE: Enhanced language representation with informative entities, in: A. Korhonen, D. Traum, L. Màrquez (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 1441–1451. URL: https://aclanthology. org/P19-1139. doi:10.18653/v1/P19-1139. [18] C. Rosset, C. Xiong, M. Phan, X. Song, P. Bennett, S. Tiwary, Knowledge-aware language model pretraining, arXiv preprint arXiv:2007.00655 (2020). [19] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t.
Yih, T. Rocktäschel, et al., Retrieval-augmented generation for knowledge-intensive nlp tasks, Advances in Neural Information Processing Systems 33 (2020) 9459–9474. [20] A. Kumar, A. Pandey, R. Gadia, M. Mishra, Building knowledge graph using pre-trained language model for learning entity-aware relationships, in: 2020 IEEE International Conference on Computing, Power and Communication Technologies (GUCON), IEEE, 2020, pp. 310–315. [21] Z. Zhang, X. Liu, Y. Zhang, Q. Su, X. Sun, B. He, Pretrain-KGE: Learning knowledge representation from pretrained language models, in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 259–266. URL: https://aclanthology.org/2020.findings-emnlp. 25. doi:10.18653/v1/2020.findings-emnlp.25. [22] J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, Y. Gong, H.-Y. Shum, J. Guo, Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph, arXiv preprint arXiv:2307.07697 (2023). [23] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al., Judging llm-as-a-judge with mt-bench and chatbot arena, Advances in Neural Information Processing Systems 36 (2024). [24] D. Stammbach, V. Zouhar, A. Hoyle, M. Sachan, E. Ash, Revisiting automated topic model evaluation with large language models, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 9348–9357. URL: https: //aclanthology.org/2023.emnlp-main.581. doi:10.18653/v1/2023.emnlp-main.581. [25] C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, J. E. Gonzalez, Memgpt: Towards llms as operating systems, 2024. arXiv:2310.08560.