The analysis of Visually-Rich Documents (VRDs) is crucial in the banking sector to support Trend and Risk Analysis (TRA) as financial TRA documents are multimodal to a large extent. Recently, Retrieval Augmented Generation (RAG) systems have enabled the efective use of Large Language Models (LLMs) to answer questions related to multimodal content. However, the inherent verbosity and complexity of financial documents could degrade the quality of the generated answers. In this work, we explore the use of text summarization techniques to condense the information retrieved from TRA-related VRDs. We analyze the level of synthesis of the original RAG answers, both with and without cascading an ad hoc summarization step. We apply summarization performance measures to compare standard RAG answers with the summarization outputs achieved on the retrieved passages directly. The results show that proprietary LLMs (GPT-4o) significantly improve the RAG's ability to sum up the retrieved passages, whereas integrating open-source LLMs or traditional summarizers turns out to be not beneficial even while applying the summarization step on top of the RAG answer.
Visually-Rich Documents (VRDs) are types of documents
that are commonly used in the banking sector to perform
Trend and Risk Analysis (TRA). They consist of visual and
textual elements such as charts, diagrams, textual
paragraphs, and tables. Multimodal elements collectively refer to
semantic entities whose identification, comprehension, and
elaboration are crucial to solve advanced reasoning tasks
such as Visual Question Answering [
In the banking sector, analysts of TRA units often need to
query financial VRDs to gain insights into the latest
advancements in economic and technological fields. To support
this time-consuming activity, the use of Large Language
Models has become increasingly appealing [
To deal with this issue, we first design and implement a RAG system to manage the financial VRDs provided by three separate TRA units of a primary banking institution. Next, given the textual passages shortlisted by the multimodal retrieval step, we compare the level of synthesis of the RAG outputs produced by three alternative strategies: (S1) Classical RAG: The LLM is prompted with the content of the retrieved passages without explicitly enforcing any summarization constraints; (S2) Summarization: The retrieved passages are summarized by an ad hoc summarization module; (S3) Cascade of RAG and Summarization: The output of S1 is summarized by an ad hoc summarization module.
We compare the summarization performance achieved by the above-mentioned strategies S1-S3 against a humangenerated ground truth. The goal is to address the following research questions: (Q1) Are LLMs efective in summarizing TRA document passages? (Q2) To what extent are RAG outputs less similar than summarizers’ outputs to ground truth summaries? (Q3) Is it beneficial to apply text summarization on top of the Classical RAG answers?
The experimental results show that the answers provided by Classical RAG to TRA-related questions are inherently redundant, calling for ad hoc summarization strategies. While proprietary LLMs excel at generating concise summaries of the retrieved passages, the level of synthesis of open-source models (including LLMs) is, in general, not satisfactory. Furthermore, cascading RAGs with summarization modules (regardless of the approach used) turns out to be not beneficial.
Given a set of Visually-Rich Documents related to Trend, Innovation, and Risk Analysis in the banking sector and a textual question ∈ on , the RAG system returns the answers to each question by performing the following steps: (1) Document chunking and encoding: it recognizes and splits the visual and textual elements in the documents’ content, generates alternative textual descriptions of the visual elements, and encodes the text corresponding to each element separately; (2) Passage retrieval: it encodes the ques
Documents
Retriever Retrieved passages Summarizer sim(RAGtP , P) sim(S-RAGtP , P) sim(StP , P) Summarizer Summary of RAG Output Summary of passages sim(RAGtP , GTt) sim(StP , GTt) sim(S-RAGtP , GTt)
Ground truth summary tion and retrieves the top- passages from that are most relevant to ; (3) LLM prompting: it prompts the LLM with both the question and the retrieved passages . Note that Classical RAG prompts are designed for Question Answering and do not include any explicit summarization step.
We aim to analyze the level of synthesis of the RAG output . Specifically, given an input question ∈ , we compare the corresponding passages and the (humangenerated) ground truth summary of with the following outputs: (1) Classical RAG: The final output of the RAG, denoted by ; (2) Summarizer: The output of an external summarizer that takes as input , denoted by ; (3) Cascade of RAG+Summarizer: The output of an external summarizer that takes as input the RAG output , denoted by - .
The diagram in Figure 1 shows the scenario under analysis, where the similarities between the retrieved passages and the outputs are depicted using blue dashed lines, whereas those between the outputs and the ground truth summaries are depicted using red dashed lines. Note that the summarizer module is not necessarily integrated into the RAG system, as we explore various summarization approaches and models, including abstractive summarization (using both LLMs and non-LLM models) and hybrid strategies combining extractive and abstractive methods.
We implement the RAG system using the LangChain framework.
elements using the proprietary Document Intelligence
service provided by the Azure AI platform [
Passage retrieval We retrieve passages via textual semantic similarity. Specifically, we retrieve the textual content associated with the elements in whose embeddings maximize the cosine similarity with the ’s encoding. LLM prompting We consider the proprietary LLM GPT4o and use the following prompt: You are a virtual assistant that can do Q&A. Try to answer without using bullet points. Given the following context, try to write a text that highlights the topics discussed in the question. If any of the context elements are not useful, ignore them. If you don’t know the answer, just say you don’t know, don’t try to invent an answer, but say that the documents you have can’t satisfy the request. [context] [question] where [context] and [question] are the retrieved passages and the current question, respectively.
External summarizers We conduct experiments with
traditional Transformer-based models, i.e., LED [
Focus on the topic of [keyword]: [to_summarize] where we replace [keyword] with the question expressed as a keyword and [to_summarize] with the corresponding retrieved passages to summarize. We also test two hybrid strategies combining extractive summarization using graph-based (TextRank, LexRank) or clustering (K-Means) methods with an LLM-based generative step.
We evaluate the pairwise textual similarities between passages, LLM answers, and summaries using the following strategies:
Open-source models are accessed via the Hugging
Face Transformers library and the proprietary GPT-4o
(gpt-4o-2024-05-13) model [
We ask TRA units’ experts to generate questions corresponding to distinct keywords (72 for ICT Risk, 19 for Innovation, and 11 for Trend). For each question, ground truth summaries are manually annotated by at least 3 units’ experts.
swer Q1, we conduct a human validation of the passage
summaries generated by the best-performing open-source
LLM according to the automatic evaluation metrics and
GPT-4o. TRA units’ experts evaluated each output as Very
bad, Bad, Moderate, Good, or Very good according to the
following facets: Grammaticality, Usefulness, Coherence,
NonRedundancy, and Overall Quality [
S-RAGtP (A) vs.
StP (B)
RAGtP (A) vs.
StP (B)
S-RAGtP (A) vs.
StP (B)
RAGtP (A) vs.
StP (B)
S-RAGtP (A) vs.
StP (B)
Q2, we evaluate the similarities between the ground truth summaries and the outputs of (1) The classical RAG (see line Output of Classical RAG in Table 2), and (2) The best configurations for each dataset of the diferent summarizers employed (see lines GPT-4o, Llama3-Instruct, Zephyr, LED large, BART large, and hybrid strategies in Table 2). The results indicate that GPT-4o outperforms Classical RAG in terms of coherence with the ground truth summaries, whereas all the other summarizers, including the open-source LLMs and hybrid approaches, generally perform on par with or even worse than Classical RAG.
Q3, the line Output of Cascade RAG+Summarizer in Table 2 reports the summarization performance of the approach based on applying summarization on top of the RAG output. In this case, we always use the best-performing model (i.e., GPT-4o) as the summarizer. The comparison with Classical RAG shows that cascading is never beneficial, even when employing the most efective summarizer, consistently resulting in performance degradation. These findings suggest that, since RAG answers are not specifically designed for the summarization task, concise answers condensing the relevant retrieved information can be produced more efectively by applying an explicit summarization step directly to the retrieved passages. Notably, summarizing the RAG output generated in the previous step fails to achieve the same quality as a direct summarization step, leading to even lower performance.
LLM-based similarity We carry out an A/B test using GPT-as-an-expert to compare Classical RAG against summarizers’ outputs (see the left-hand side bars of the plots in Figure 2) and Cascade RAG+Summarizer against summarizers’ outputs (see the right-hand side bars). We consider as summarizer GPT-4o, the best-performing open-source LLM and traditional Transformer-based model separately for each dataset. The results align with the automatic evaluation: GPT-4o as a summarizer outperforms both Classical and Cascade RAG+Summarizer (>80% vs. <20%), opensource LLMs perform comparably with them (both around 50%), whereas non-LLM summarizers demonstrate worse performance (<20% vs. >80%).
Comparison between RAG and summarizers’ outputs with respect to retrieved passages Instead of evaluating the similarities between the generated outputs and the ground truth summaries (see Table 2), in this analysis we consider the retrieved passages as references. Specifically, we report in Table 3 the results of the comparisons between the retrieved passages and the outputs of (1) The classical RAG (see line Output of Classical RAG), (2) The cascade of RAG and summarizer (see line Output of Cascade RAG+Summarizer), and (3) The best configurations for each dataset of the diferent summarizers tested (see lines GPT-4o, Llama3-Instruct, Zephyr, LED large, BART large, and hybrid strategies).
In most cases, GPT-4o outperforms all the other approaches including open-source LLMs, traditional Transformer-based models, and hybrid strategies. Notably, it achieves significantly higher performance compared to both types of RAG outputs (i.e., Classical and Cascade RAG+Summarizer). Similar to the previous analysis, applying an additional summarization step on top of the RAG output proves to be detrimental, leading to even lower scores across all metrics. The results are aligned with those obtained using ground truth summaries as references. However, here we focus on a diferent aspect. Higher similarity with respect to the retrieved passages denotes better attribution of the generated text to the source passages. This is particularly relevant in TRA domains, where maintaining a high level of accountability to the document sources is critical. In conclusion, both sets of results indicate that GPT-4o, when used as a summarizer directly applied to the retrieved passages, excels at generating summaries that align well not only with the ground truth summaries but also with the source documents. In contrast, the other summarizers, and in particular the two types of RAG outputs considered, demonstrate lower performance.
This paper explored methods for summarizing passages retrieved by a RAG system indexing financial VRDs related to Trend and Risk Analysis in the banking sector. Employing proprietary LLMs as summarizers enhances the level of synthesis of classical RAG outputs, whereas open-source LLMs or traditional summarizers do not achieve significant performance improvements due to the inherent complexity of multimodal, domain-specific sources. Notably, applying summarizers directly to the retrieved passages has shown to be more efective than cascading RAGs with an additional summarization step.
As future work, we plan to generate summarized answers
using RAG systems with diferent characteristics and
evaluate them on existing benchmarks, as well as using
sequenceto-sequence models specialized on languages other than
English [
We identify the following limitations of our work: Open-source LLMs Due to computational constraints, we have currently tested the 8B parameter version of Llama3Instruct. As a future extension, we plan to evaluate a broader suite of open-source LLMs with varying levels of complexity. It is worth noting that, despite their significantly smaller number of parameters, the open-source LLMs considered show fairly good performance, in some cases comparable to that of larger, proprietary ones.
Model fine-tuning Currently, both LLMs and traditional Transformer-based model versions we employ are not specialized on domain-specific data. We plan to fine-tune a selection of models to generate more domain-aware summaries.
RAG architecture For visual elements, the retrieval
module currently relies on semantic similarity between textual
descriptions of multimodal elements generated using
GPT4o. We aim to explore the use of diferent Multimodal LLMs
that also capture layout information (e.g., LayoutLLM [
Proprietary LLMs are, by design, non-transparent, therefore the reproducibility of their results is limited and the motivations behind their weaknesses remain partly obscure.
LLMs are also known to sufer from bias issues. Although we verified the coherence of the summarization outputs with the help of domain experts, we cannot guarantee that the summaries produced by generative AI models in a realworld scenario are entirely free of hallucination. This could be mitigated by introducing ad hoc approaches or models to evaluate the factuality of the generated answers, enabling to filter out or possibly improve non-factual outputs.
The proprietary data were selected by banking units’ experts who were explicitly trained on how to properly sample the document collections under analysis. While the overall quality of the data sampling is above average, we cannot exclude the possibility that the collection may contain outliers or minor errors.
Documents cannot be disclosed due to confidentiality and copyright restrictions. Code could be made available upon request to the authors.
Intesa Sanpaolo is a leading banking group in the Eurozone and the largest one in Italy. Intesa Sanpaolo Innovation Center is part of ISP group, and its mission is exploring business models of the future to discover new assets and skills that support the long-term competitiveness of ISP group and its customers. ISP has established the Innovation Center Labs to respond to the complex needs of the bank and the market, determined by the evolution of market trends and exponential technology growth.
The authors would like to thank Giorgio Bella, Anna Polise, Stefania Vigna, Laura Li Puma, Chiara Napione, Giovanni Troiano, Patrizio Paolo Dionisi, Maura Bertaglia, Carla Monferrato, and Simone Scarsi for their useful comments. They would also like to thank Luigi Ruggerone for supporting the research team.
The work by Giuseppe Gallipoli was carried out within the MICS (Made in Italy – Circular and Sustainable) Extended Partnership and received funding from NextGenerationEU (Italian PNRR – M4 C2, Invest 1.3 – D.D. 1551.11-10-2022, PE00000004). This study was also partially carried out within the FAIR (Future Artificial Intelligence Research) and received funding from Next-GenerationEU (Italian PNRR – M4 C2, Invest 1.3 – D.D. 1555.11-10-2022, PE00000013). This manuscript reflects only the authors’ views and opinions, neither the European Union nor the European Commission can be considered responsible for them.