Refinement for RAG

Abstract

Recent advancements in the contextual understanding and generation capabilities of Large Language Models (LLMs) have sparked increasing interest in the application of Retrieval-Augmented Generation (RAG) in specific domains and industry documents. Retrieving and understanding tables within these documents is crucial for generating correct answers in RAG systems. This study focuses on documents containing large and complex tables, such as statistical and industry reports and these presents two major challenges: 1) processing the large tables and 2) understanding complex tables. Previous studies faced challenges as they considered elements of tabular data such as cells, headers, and titles. In contrast, we designed the Table Header for Retrieval and Refinement (THoRR) method to address the aforementioned issues. THoRR performs two tasks: table retrieval and table refinement. In the table retrieval phase, we propose a table header representation approach that uses headers and titles, without considering cells. In the refinement phase, the model selects relevant table headers from the retrieved tables and processes them into refined tables containing the necessary information to answer the questions. This approach aids in understanding complex tables without chunking, by reorganizing information. Our models outperform existing approaches such as DTR and DPR-table. Moreover, we experimentally demonstrate that our refinement model can reduce hallucinations. To the best of our knowledge, our table refinement approach for RAG system is the first of its kind in the field. table retrieval, complex table, retrieval-augmented generation (RAG), table refinement, table representation Workshop Proceedings

1. Introduction

The first challenge involves processing large and complex were designed with relatively simple open-domain tables in mind, such as those found in the nq-table[ 5 ] dataset, thus de-emphasizing the processing of large tables. Similar to the processing of text documents, previous methods involved dividing data tables into fixed-length segments (chunking), or even cutof parts that exceeded a maximum input length. The chunking method complicates data retrieval by not only increasing the number of retrieval targets but also making it challenging to compare values across segmented tables. Moreover, disregarding overflow sections risks losing table information, diminishing the probability of obtaining a sufifcient table representation. These problems can ultimately afect table retrieval performance.

The second challenge is the dificulty in understanding tables due to their complex structure. Complex tables typically feature hierarchical headers and numerous values, presenting a challenge for generator to consider vast amounts of information. Insuficient table comprehension can lead to incorrect answers (hallucinations). Figure 1 demonstrates an example where GPT-3.5-turbo[ 6 ] is used to perform tableQA on a hierarchical table. It showcases how the original table leads to incorrect responses, whereas the refined table, as processed by our proposed model, yields the correct answers.

In this paper, we propose Table Header For Retrieval and Refinement ( THoRR) to solve this problem. These method is grounded in a heuristic assumption that, when finding and understanding tables, headers are more critical than values. THoRR has two models, a retriever and a refinement model, performed sequentially. Each is diferent from the previous one. THoRR: Retriever uses a table header representation. It performs table retrieval using only the header without considering the cells of the table. THoRR: Refinement performs relevant table header detection in the retrieved table to select table headers that are relevant to the question, and refines them into a simple table that contains only the necessary information, reducing the amount of information the generator needs to consider.

We compare THoRR with DTR[ 3 ] and DPR-table[ 4 ] and show that it has better retrieval performance in fine-tuning and zero-shot experiments on the HiTab[ 7 ] and AIT-QA[ 8 ] the result of the original complex table (top) and the refined table (bottom).

Recent advancements in the contextual understanding and generative capabilities of Large Language Models (LLMs) have heightened interest in Retrieval-Augmented Generation (RAG)[ 1, 2 ] for specific domains such as open domain or industry-specific documents. Industry or finance domain documents often contain large and complex tables. The understanding of which is critical for a RAG system to produce accurate responses. However, this task presents several challenges. Our research seeks solutions to two primary challenges.

IR-RAG’24: Information Retrieval’s Role in RAG Systems (IR-RAG), July ∗Corresponding author.

0009-0005-9453-7443 (K. Kim)

CEUR

ceur-ws.org datasets while reducing the information (number of cells) needed to input the generator. Furthermore, our proposed methodology enables an eficient reduction in the number of tokens required for table inputs in the generator. 2.

Method

In this section, we present the Table Header For Retrieval and Refinement ( THoRR) method, designed to retrieve and refine tables within the RAG system [ 2 ]. THoRR is divided into two phases, retrieval and refinement, as shown in Figure 2. These two phases are separately trained and serve distinct purposes. The retrieval phase is utilized for embedding and indexing tables. Subsequently, as a question is input, it retrieves the pre-indexed tables. In the refinement phase, the retrieved tables are processed to extract the necessary information, refining them into smaller tables.

The goal of this method is to obtain the _ refined Tables relevant to the given target Tables when a question is provided. We denote the components of as , ℎ , and ℎ

, representing the row headers, column headers, and title, respectively. The comparative experiments between THoRR and the existing table retrieval baseline are explained in Section 3.1

2.1. Table Retriever

Given M target tables T, Our THoRR:retrieval model aims to retrieve the

_ candidate tables containing information relevant to the question Q. In this paper, we follow the structure of DPR[ 9 ] for comparison with DPR-table[ 4 ]. we use two diferent encoders (the table header encoder ( the question encoder ( of [ 10 ]. maps target ), both utilizing the base model , and then select the

_ closest candidate tables for indexed from it. The similarity between and is defined by using the dot product, as in [ 9 ] (equation 2), and the encoder uses the base model of [ 10 ].

= {[] [ ] ℎ [ ] ℎ [ ]} (, ) = () ⊤ ⋅ ( ) (1) (2) In this process, a diference aspect of our retriever compared to previous research lies in the table header representation

2.2. Table Refinement Model

equation 4, where the ℎ ∈ [ℎ This paper introduces a new task called Table Refinement, defined as simplifying a table while preserving specific information. Accordingly, our THoRR:refinement model aims to obtain refined tables, denoted as , for the , ℎ ]. Similar to equation 5, is input to the refinement encoder to obtain hidden states. And then, the linear layer takes in these hidden states and outputs the relevant header score, denoted as ℎ. Using ℎ, we obtain the _ relevant column headers indices ( ) and

_ relevant row header indices ( ) as specified in Equation 6. Subsequently, we refine candidate tables using selected row and column indices to obtain . (3) (4) (5) (6) (7) . Our method, which utilizes the table’s header and title without considering every cell, is relatively free from the input limitations of the encoder. The chunking method and our comparative experiments are explained in Section 3.2

The objective of the training is to minimize the disas Negative Log Likelihood (NLL) : tance between questions q and positive table maximizing the distance between queries and the number of negative tables = {( , + , − ,1 , − ,2 , ..., − , )}

− in a given training dataset =1 . The loss function, optimized + while ( , + , − ,1 , − ,2 , ...,

,− ) = −

∑=1 ( (

, +) , +) + ( , ,−) = {[] [ ] ℎ [ ]} ℎ = ( ) = (ℎ = (ℎ , , _) _) and is the gold relevant header index.

The learning objective aims to identify the index of the question and relevant header, with the goal of increasing the score of the answer’s header index ℎ . The loss function is as described in Equation (5), We optimized Cross Entropy Loss. Where, represents the number of tokens in input (ℎ, ) = −

∑=1 (ℎ

) = (ℎ )

AIT-QA Zero-shot HIT@5 HIT@10 HIT@20

3. EXPERIMENTS

Dataset We conduct experiments on two complex table benchmark datasets. HiTab[ 7 ] is a Table QA dataset with a hierarchical structure. This dataset consists of questions that require complex numerical calculations, including tables from Wikipedia and statistical reports. It contains a total of 10,672 question-answer pairs, with 7,417 for training, 1,671 for validation, and 1,584 for testing. There are a total of 3,597 tables in this dataset. We use this dataset for fine-tuning. AIT-QA[ 8 ] is a Table QA dataset specific to the Airline industry, composed of tables extracted from the U.S. public SEC filings. It includes specialized vocabulary terms for a specific domain and also has a hierarchical structure like HiTab[ 7 ]. It consists of 515 questions and answers, with a total of 116 tables. In this paper, this dataset is used to evaluate the zero-shot performance of the fine-tuning model.

Baseline In order to demonstrate the performance of our method, we compare it with baseline methods. DTR[ 3 ] is a table encoder that uses a table-specific structure. DPRtable[ 4 ], on the other hand, processes tables linearly, similar to understanding text passages. Both of these baselines have been trained on the nq-dataset[ 5 ] and their pretrained models are publicly available. We fine-tune these pre-train models as backbones and compare them with our model.

3.1. Main Result : THoRR

The experiments in this paper evaluate the proposed models, THoRR, in a two-phase process as shown in Figure 1 (THoRR:retrieval and THoRR:refinement). The performance of the models is evaluated using the ’Hits accuracy’ as the main evaluation metric. This metric measures the ratio of correct answers included in the _ selected tables by the models. Where, _ takes values 1, 5, 10, 20, 50 to evaluate the accuracy of the models.

Fine-tuning To compare fine-tuning experiments on the complex table dataset, we train THoRR and baselines using the HiTab [ 7 ] training set. Table 1 presents the performance of the THoRR method compared to baseline models. The experimental results indicate that the proposed models outperformed baselines in most cases. When _ = 7 and _ <= 10 , the proposed models exhibit an accuracy improvement of more than 5% compared to the baseline’s best accuracy. The superior performance at a small _ indicates the importance in the RAG system, as it indicates efective utilization of a limited number of reference pieces of information, which is common when the _ is less than 10.

Zero-shot The zero-shot experiment intend to observe how the model performs on complex table data from a new domain. In this process, a fine-tuned model using the HiTab [ 7 ] dataset is used to make predictions on the AIT-QA[ 8 ] dataset without any additional training, and the results are evaluated. Through this experiment, we aim to demonstrate that the proposed models can handle complex table retrieval in previous unseen domains. Table 1 presents the results of this experiment, showing superior performance compared to the baselines and indicating well THoRR works on complex tables in diferent domains.

3.2. Retrieval Result

(a) (b)

We compare our proposed table header representation method and chunking method in terms of retrieval accuracy. Figure 3(a) illustrates the performance of [ 4 ] with the chunking method and our method. (”inf” refers to the use of the original table without chunking.) As shown in Figure 3(b), we observe a decrease in retrieval accuracy as the lower max token length, indicating that the number of retrieval targets afects the performance significantly in retrieval tasks. Our approach demonstrates superior performance compared to methods that consider all values. This highlights the efectiveness of our method, which relies solely on table headers for table representation, especially in retrieving large and complex tables. Moreover, our method demonstrates superior performance compared to existing approaches that consider all values, thereby experimentally validating our heuristic assumption that headers are crucial elements in table retrieval.

3.3. Refinement Result

In this section, we experiment with our refinement model to reduce cell information in mitigating hallucinations. In Figure 4, the green line indicates a decreasing trend in the number of cells in tables when using our model. Furthermore, Figure 4 illustrates the human evaluation accuracy on the results obtained by input refined tables into Llama2[ 11 ] 7B-Chat. Where, ”(-,-)” denotes the original table. We randomly sample 300 questions from the HiTab test dataset for human evaluation. Llama2[ 11 ] 7B-Chat takes a gold table as input to generate responses. If the generated response contains exactly the answer and is correct, we mark it as correct. Otherwise, we consider it as a hallucination. Three master’s students in the field of AI evaluated the generated results. To ensure the reliability of the evaluations, one evaluator and two validators were assigned roles in the evaluation process. As a result, by setting _ = 7 and _ = 10 , we demonstrate that our refinement model reduces the number of table cells from 153.88 to 58.03, resulting in a 62.2% decrease compared to the original table. Additionally, we observe a 9.33% improvement in the reduction of hallucinations. This validates the superiority of our refinement approach.

4. Related Works

Research on table encoders has been focused on pre-training tabular data with table-specific architectures[ 12, 13, 14, 15, 16, 17 ]. TAPAS[ 12 ] introduces a pre-training method using Masked-Language-Modeling for the cells of tabular data. TaBERT[ 13 ] introduces a pre-training model that jointly learns over 26 million natural language questions and tables. TURL[18] introducing a structure-aware Transformer encoder and Masked Entity Recovery (MER) objective for pre-training. StruG[ 14 ] proposes a semi-supervised learning framework for learning the connection between text and SQL. MATE[ 15 ] demonstrates the eficient restriction of Transformer attention flow on tabular data, enabling training with larger sequence lengths. Tableformer[ 16 ] learns from tables using attention biases, making it better at understanding tabular data. TABBIE[17] introduces a method to improve performance on table-based prediction tasks by pre-training only tabular data.

Research on table retrieval includes methodologies such as [ 19, 3, 4, 20, 21 ]. Table2vec[19] proposes a method for obtaining table embeddings by considering various table elements such as captions, headers, cells, and entities. DTR[ 9 ] introduces a table-specific model suitable for open-domain table question answering. DPR-table[ 4 ] linearizes tables to handle them similar to text passages, instead of using table-specific models. GTR[ 20] introduces a model that transforms tables into graphs, capturing both cell and layout structures. [21] introduces a method for enhancing the similarity between queries and tables for table retrieval, employing various semantic spaces and similarity measurement methods.

5. Conclusion

We propose the THoRR method, which uses the table headers to retrieve and help understand the complex and large tables. We use the table header representations in the retriever that can retrieve tables without chunking them. Additionally, we propose a novel methodology for refining tables by detecting the table headers that are relevant to the questions within the table. This approach aims to simplify the tables in which an excessive amount of information is present, particularly in complex tables. THoRR is capable of handling large and complex tables without dividing them into smaller chunks, reducing the information required for preventing hallucinations in LLM generator. Furthermore, the Table Refinement task is the first of its kind in this field, therefore, it is expected to contribute significantly to the future research in this field. Our future work involves exploring methods to detect the table headers. Additionally, we aim to prevent potential information loss in questions by selecting fewer relevant headers during the refinement phase. 18653/v1/2022.acl-long.40. [17] H. Iida, D. Thai, V. Manjunatha, M. Iyyer, TABBIE: Pretrained representations of tabular data, in: K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, Y. Zhou (Eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 3446–3456. URL: https://aclanthology.org/2021.naacl-main.270. doi:10. 18653/v1/2021.naacl-main.270. [18] X. Deng, H. Sun, A. Lees, Y. Wu, C. Yu, Turl: table understanding through representation learning, Proc. VLDB Endow. 14 (2020) 307–319. URL: https://doi.org/10.14778/3430915.3430921. doi:10. 14778/3430915.3430921. [19] L. Zhang, S. Zhang, K. Balog, Table2vec: Neural word and entity embeddings for table population and retrieval, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 1029–1032. URL: https://doi.org/10.1145/3331184.3331333. doi:10. 1145/3331184.3331333. [20] F. Wang, K. Sun, M. Chen, J. Pujara, P. Szekely, Retrieving complex tables with multi-granular graph representation learning, SIGIR ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 1472–1482. URL: https://doi.org/10.1145/3404835.3462909. doi:10. 1145/3404835.3462909. [21] S. Zhang, K. Balog, Semantic tablenbsp;retrieval using keyword and table queries, ACM Trans. Web 15 (2021). URL: https://doi.org/10.1145/3441690. doi:10. 1145/3441690.

References

[1] P. Lewis , E. Perez , A. Piktus , F. Petroni , V. Karpukhin , N. Goyal , H. Küttler , M. Lewis , W.-t. Yih, T. Rocktäschel , S. Riedel , D. Kiela , Retrieval-augmented generation for knowledge-intensive nlp tasks , in: Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Curran Associates Inc., Red Hook , NY , USA, 2020 .

[2] Y. Gao , Y. Xiong , X. Gao , K. Jia , J. Pan , Y. Bi , Y. Dai , J. Sun , Q. Guo , M. Wang , H. Wang , Retrievalaugmented generation for large language models: A survey , ArXiv abs/2312 .10997 ( 2023 ). URL: https: //api.semanticscholar.org/CorpusID:266359151.

[3] J. Herzig , T. Müller , S. Krichene , J. Eisenschlos , Open domain question answering over tables via dense retrieval , in: K. Toutanova , A. Rumshisky , L. Zettlemoyer , D. Hakkani-Tur , I. Beltagy , S. Bethard , R. Cotterell , T. Chakraborty , Y. Zhou (Eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics , Online, 2021 , pp. 512 - 519 . URL: https://aclanthology.org/ 2021 .naacl-main. 43 . doi: 10 . 18653/v1/ 2021 .naacl- main.43.

[4] Z. Wang , Z. Jiang , E. Nyberg, G. Neubig, Table retrieval may not necessitate table-specific model design , in: W. Chen , X. Chen , Z. Chen , Z. Yao , M. Yasunaga , T. Yu , R. Zhang (Eds.), Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI) , Association for Computational Linguistics , Seattle, USA, 2022 , pp. 36 - 46 . URL: https://aclanthology.org/ 2022 .suki- 1 .5. doi: 10 .18653/v1/ 2022 .suki- 1 .5.

[5] T. Kwiatkowski , J. Palomaki , O. Redfield , M. Collins, A. Parikh , C. Alberti , D. Epstein , I. Polosukhin , J. Devlin , K. Lee , K. Toutanova , L. Jones , M. Kelcey , M.- W. Chang , A. M. Dai , J. Uszkoreit , Q. Le , S. Petrov , Natural questions: A benchmark for question answering research, Transactions of the Association for Computational Linguistics 7 ( 2019 ) 452 - 466 . URL: https: //aclanthology.org/Q19-1026. doi: 10 .1162/tacl_a_ 00276 .

[6] T. B. Brown , B. Mann , N. Ryder , M. Subbiah , J. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry , A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger, T. Henighan , R. Child , A. Ramesh , D. M. Ziegler , J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler, M. Litwin , S. Gray , B. Chess , J. Clark , C. Berner , S. McCandlish , A. Radford , I. Sutskever , D. Amodei , Language models are few-shot learners , in: Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Curran Associates Inc., Red Hook , NY , USA, 2020 .

[7] Z. Cheng , H. Dong , Z. Wang , R. Jia , J. Guo , Y. Gao , S. Han, J .-G. Lou, D. Zhang, HiTab: A hierarchical table dataset for question answering and natural language generation , in: S. Muresan, P. Nakov , A . Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers) , Association for Computational Linguistics , Dublin, Ireland, 2022 , pp. 1094 - 1110 . URL: https://aclanthology.org/ 2022 . acl-long . 78 . doi: 10 . 18653/v1/ 2022 .acl- long.78.

[8] Y. Katsis , S. Chemmengath , V. Kumar , S. Bharadwaj , M. Canim , M. Glass , A. Gliozzo , F. Pan , J. Sen , K. Sankaranarayanan , S. Chakrabarti, AIT-QA: Question answering dataset over complex tables in the airline industry , in: A. Loukina , R. Gangadharaiah , B. Min (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, Association for Computational Linguistics , Hybrid: Seattle, Washington + Online, 2022 , pp. 305 - 314 . URL: https://aclanthology. org/ 2022 .naacl-industry. 34 . doi: 10 .18653/v1/ 2022 . naacl- industry.34.

[9] V. Karpukhin , B. Oguz , S. Min , P. Lewis , L. Wu , S. Edunov , D. Chen , W.-t. Yih, Dense passage retrieval for open-domain question answering , in: B. Webber , T. Cohn, Y. He , Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Association for Computational Linguistics , Online, 2020 , pp. 6769 - 6781 . URL: https://aclanthology.org/ 2020 .emnlp-main. 550 . doi: 10 .18653/v1/ 2020 .emnlp- main.550.

[10] J. Devlin , M.- W. Chang , K. Lee , K. Toutanova , BERT: Pre-training of deep bidirectional transformers for language understanding , in: J. Burstein , C. Doran , T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers), Association for Computational Linguistics , Minneapolis, Minnesota, 2019 , pp. 4171 - 4186 . URL: https://aclanthology. org/N19-1423. doi: 10 .18653/v1/ N19 - 1423.

[11] H. Touvron , L. Martin , K. Stone , P. Albert , A. Almahairi , Y. Babaei , N. Bashlykov , S. Batra , P. Bhargava , S. Bhosale , et al., Llama 2 : Open foundation and finetuned chat models , arXiv preprint arXiv:2307.09288 ( 2023 ).

[12] J. Herzig , P. K. Nowak , T. Müller , F. Piccinno , J. Eisenschlos, TaPas: Weakly supervised table parsing via pre-training , in: D. Jurafsky , J. Chai , N. Schluter , J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Online, 2020 , pp. 4320 - 4333 . URL: https: //aclanthology.org/ 2020 .acl-main. 398 . doi: 10 .18653/ v1/ 2020 .acl- main.398.

[13] P. Yin , G. Neubig, W.-t. Yih, S. Riedel, TaBERT: Pretraining for joint understanding of textual and tabular data , in: D. Jurafsky , J. Chai , N. Schluter , J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Online, 2020 , pp. 8413 - 8426 . URL: https:// aclanthology.org/ 2020 .acl-main. 745 . doi: 10 .18653/ v1/ 2020 .acl- main.745.

[14] X. Deng , A. H. Awadallah , C. Meek , O. Polozov , H. Sun , M. Richardson , Structure-grounded pretraining for text-to-SQL, in: K. Toutanova , A. Rumshisky , L. Zettlemoyer , D. Hakkani-Tur , I. Beltagy , S. Bethard , R. Cotterell , T. Chakraborty , Y. Zhou (Eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics , Online, 2021 , pp. 1337 - 1350 . URL: https://aclanthology.org/ 2021 .naacl-main. 105 . doi: 10 . 18653/v1/ 2021 .naacl- main.105.

[15] J. Eisenschlos , M. Gor , T. Müller , W. Cohen, MATE: Multi-view attention for table transformer eficiency , in: M. - F. Moens , X. Huang , L. Specia , S. W.-t. Yih (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Online and Punta Cana , Dominican Republic, 2021 , pp. 7606 - 7619 . URL: https://aclanthology.org/ 2021 .emnlp-main. 600 . doi: 10 .18653/v1/ 2021 .emnlp- main.600.

[16] J. Yang , A. Gupta , S. Upadhyay , L. He , R. Goel , S. Paul, TableFormer: Robust transformer modeling for tabletext encoding , in: S. Muresan, P. Nakov , A . Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers) , Association for Computational Linguistics , Dublin, Ireland, 2022 , pp. 528 - 537 . URL: https://aclanthology.org/ 2022 . acl-long . 40 . doi:10.