Open-domain QA has arisen as a natural language processing research subject to answer users' inquiries in natural language from vast unstructured text collections. Among the most important business areas, using a corpus of unstructured documents as a source of information to answer a question is just one of the many benefits of working in an open-domain setting, which also allows for access to external knowledge even if the information related to the question is not uniquely identified in the sources. Thus, any procedures involving extracting information and data from available online sources and text documents may be enhanced with the help of open-domain question-answering systems. In this paper, we propose a method for developing an open-domain question-answering system by retrieving documents from a knowledge corpus comprising external and internal documents in response to input query and then extracting the answer using an instruction fine-tuned language model, following a “zero-shot” approach. While conceptually simple, this approach can be used as a flexible framework to develop an open-domain question-answering system eficiently.
nlp, open domain question answering, language models, information retrieval information in response to an input query is retrieved. learning behavior [11] that allows them to perform
Open-domain
question answering (OpenQA) has emerged as a natural language processing (NLP) research ifeld aimed at answering questions in natural language from retrieved extensive unstructured text collections in response to users’ questions. In leading business domains, an open-domain setting allows using a corpus of unstructured texts to answer a query and access external knowledge, even if the information related to the query is not univocally identified in the sources.
enhance data extraction from open web sources and text documents (e.g., internal documents, emails, online reviews, and other sources). Most of the existing OpenQA systems [1], [2], [3], [4], [5], [6] follow two-stage
the question from the retrieved corpus of documents.
a knowledge corpus drawn from both open sources and internal documents to answer input questions, and then, extracting the answer using a modern instruction ifne-tuned language model.
Developing an OpenQA system involves several challenges. When you have confidentiality about the retrieved results [ 7], [8], [9]. Given an initially retrieved passage list, the reranker component scores them, enabling the selection of the most suitable answer aggregating evidence from multiple passages. When developing our system, we employed a fine-tuned instruction language model as an alternative, delegating both the question answering and aggregation-answering tasks to it. It has been proven that modern language models, particularly large ones with billions of parameters, are competitive in many NLP tasks [10], mainly due to their in-context several tasks by conditioning on input-output examples, without optimizing any parameters or performing any gradient updates. Since the employment of models with billions of parameters are both time-consuming and costly to train and query, we opted for a pre-trained instruction-finetuned language models (flan-t5-large [12]) used with a “zero-shot” strategy, meaning no demonstrations are provided at inference time.
Our approach proposes an OpenQA pipeline consisting of three steps. In the first step, the retriever component uses transformer models to retrieve documents relevant to a specific input question. In the second step, the reader component is realized with an instruction fine-tuned lanseveral prompts can be used to perform multiple natural language processing tasks including question-answering. In the last step, the language model is again used for aggregate multiple answers into one, but with diferent instructions to allow it to combine the previous answers. Overall, our contributions are as follows: • 1) We propose an OpenQA framework that utilizes a “zero-shot” approach by using transformer models in both the retriever and reader • 2) The approach is not domain-specific, but can be easily adapted to diferent use cases and models can be trained on a specific dataset to achieve better performance.
Modern OpenQA systems use a “Retriever-Reader” ap- models have also demonstrated good performance in proach: a retrieval component finds relevant passages open-domain question answering by using stored inforfrom a large corpus and a reader extracts the answer mation from pre-trained phase, without accessing any using machine reading comprehension. As far as we external knowledge or context [16]. However, they reknow, one of the earliest systems using this approach veal that such language models can only perform as well was suggested by Chen et al. in 2017 [13] with a docu- as recent OpenQA systems if they are suficiently large ment retrieval Question Answering (DrQA) system using (over 11 billion parameters), making it dificult to cona tf-idf retriever and recurrent neural network reader to stantly update them with new knowledge. Taking this match similar documents related to a query and deliver into account, Izicard et al. in 2021 put forth a retrievalthe final answer. The positive outcomes of this approach reader approach [4] that utilizes a retriever similar to the have resulted in an emphasis on the connection between previous DrQA[13], followed by a reader made up of a OpenQA performance and the retriever’s capacity to lo- generative model T5 that is instruction-tuned on several cate pertinent text excerpts for a given query. As a result, tasks, including question answering. Recently, in 2022, Wang et al. [7] added a “ranker” to improve retrieved an other OpenQA system has been proposed using a reoutcomes by ranking passages. In 2019, several works triever based approach, followed by powerful generative proposed improvements to this architecture. Wei Yang language models such as BART [5], reaching the current et al. [6] suggested a system consisting of an Anserini state-of-the-art on OpenQA benchmarks 1. retriever, a bag-of-words retriever that directly identifies text segments from Wikipedia, and a BERT reader to per- 3. Method form extractive QA by selecting the text span where the answer can be found. Seo et al. proposed a Dense-Sparse Our proposed OpenQA system involves a multi-stage Phrase Index [2] as a retriever based on a transformer retriever that uses an information retrieval system to BERT embedding of each sentence to encode syntactic collect a large corpus from open-source and internal docor semantic information of the phrase with respect to its uments. Then, in the second stage, an index is created context. This leads to an improvement in eficiency and to enable the eficient search of pertinent documents diperformance compared to the previous system DrQa[13], viding each document into multiple text chunks, and suggesting that using the most recent transformer model then converting them into semantic vectors using mod[14] is better for a retriever system. In this scenario, in ern Sentence Transformers models [17]. Each retrieved the 2020 Guu et al. introduced REALM [15], a novel text is ultimately sent to a multi-stage reader that uses pre-trained BERT language model enhanced with exter- a fine-tuned Flan-T5 [ 12] instruction model to first get nal knowledge from a retriever. The paper shows how multiple articulated answers and, then, aggregating them this improvement is mainly attributed to a pre-training task on a large text corpus, indicating that large lan- 1https://paperswithcode.com/sota/ guage models could perform well on OpenQA. Language open-domain-question-answering-on-kilt-2 require excessive computation and ultimately decrease performance. Figure 2 illustrates the retriever component as divided into three main parts further detailed: text splitting, which divides long documents into smaller parts using a sentence-based approach; embedding, which uses an AI model to convert each text into a semantic vector; and top_k retrieval, which retrieves the top_k texts most similar to the input question based on the calculated similarity between the question and each document vector.
To create numerical vectors that represent text semantics, we used transformer models [14] based on Seo et al.’s approach [2]. Because these models have a maximum input length, longer texts must be divided into smaller parts, which can disrupt meaning and make analyzing long-term dependencies dificult. While the authors of the Dense Phrase Index divided each text into phrases [2], we grouped sentences until the maximum input length was exceeded. To split text into smaller parts for deep learning models, we use Python’s sentence-splitter library 4. Sentences are grouped together until the tokenized maximum input length is reached, creating one chunk. The second chunk keeps 50% of the sentences from the first chunk and adds more until the maximum input length is reached. This process is repeated for the whole document. Although it generates more chunks and redundancy, it helps preserve context. together creating the final answer. The Figure 1 outlines our system break it down into two main components: the information retrieval system and the OpenQA system itself.
The information retrieval system gathers data from internal and external sources. We simply discuss how the system was developed since our work concentrates on the AI core. The information retriever is an external system containerized with Docker, accessible by FastAPI 2, and able to scrape various document kinds (pdf, html, word, etc.) and recover a full corpus of documents from the web. Specifically, we implemented diferent scrapers considering diferent sources (e.g., google.com) and using 3.1.2. Embedding the Scrapy 3 framework. Using the information retriever for evaluation would have been problematic due to the We use a pre-trained Sentence Transformer model [17] constantly changing document corpus. To ensure a better to encode text into a fixed-length vector. These models evaluation of the question answering task, we kept the are based on a BERT Transformer [18] and fine-tuned document corpus fixed, as explained in the Evaluation for asymmetric semantic search. The msmarco-distilbertsection. base-v4 model by Hugging Face 5 has shown the best
The OpenQA system can handle natural language generalization ability, as it was trained on the msmarco queries and provide answers based on a given corpus of corpus 6 that includes Bing questions and answers. We text documents from the previous information retrieval did not re-train a new model for a specific dataset in this system. In this paper, we will focus solely on this sys- first phase, as we aimed to obtain a generic system that tem, as it uses artificial intelligence models and can be performs well for diferent types of documents. Therecompared to the “retriever-reader” architecture used by fore, we used the sentence embedding model with frozen modern OpenQA systems. The specific retriever and weights to encode each document and the input question. reader implementations are detailed below.
The retriever identifies relevant documents from a large corpus based on an input question. It’s main objective is to filters out irrelevant texts, as applying questionanswering models directly to each input document would
3.1.3. Top_k retrieval In order to find the most similar documents to a given query, a numerical vector from the query is created and then the cosine similarity metric is applied to identify the most similar documents from the corpus. To speed up 4https://github.com/mediacloud/sentence-splitter 5https://huggingface.co/sentence-transformers/ msmarco-distilbert-base-v4 6https://microsoft.github.io/msmarco/ the search, the Faiss library 7 from Meta is used, which is eficiently implemented in C and allows for faster retrieval of the top_k most similar documents based on a cosine similarity metric. Once this step is completed, the semantic search is concluded, and the top_k texts relevant to the specified question are returned.
The system’s objective is to answer a specific question using a given list of input text. The answer can be lo- Figure 3: The full question-answering pipeline. Firstly, the cated within one or multiple input texts or may not be input prompt of the language model is utilized to input both present. The system generates an answer considering the text chunk and the associated question to generate a wellall the relevant texts retrieved, and if it is not present, aargtgicrueglaatteeddrienstpooanssein.gSleecocnodnltye,xat,llatnhde tgheenelarantgeudaagnesmweordsealries will return an empty answer. Unfortunately, we can- then prompted to generate a singular answer that takes into not combine all the texts due to the model’s maximum account all of the generated responses. input length constraint, and we cannot increase the maximum input length by opting for a bigger model, as this would sacrifice the system’s eficiency and require noncommodity hardware. Therefore, we consider each text 3.2.2. Answer aggregation as a separate input because the retriever’s previous splitting process ensured that the maximum length of each text was not exceeded. To perform the task, we chose a sequence-to-sequence model like Glass et al. in their work [5]. However, our choice fell on FLAN-T5-large [12] released on Huggingface 8 as it was fine-tuned on several instructions allowing to handle various natural language processing tasks and even tasks that it has yet to be specifically trained for. As shown in figure 3, the flant5 model is utilized twice with two distinct instructions: one for generative question answering, which extracts a detailed response from the given text, and the other for an answer aggregation task, which extracts a single final answer from the previous responses.
This step will combine the previous answers into a single response. To do this, we will concatenate all the answers together to create a single input text for the model. Although the procedure is similar to the previous step, the instructions difer as we require a precise and singular response. In this case, we have identified a prompt for the generation model: “ A n s w e r w i t h a s f e w w o r d s a s p o s s i b l e p r e c i s e l y t o t h e q u e s t i o n . $ c o n t e x t : . . . $ q u e s t i o n : . . . a n s w e r : . . . ”
At each step, the $ c o n t e x t is replaced with the concatenation of all the previous answers and the $ q u e s t i o n with the real question. The generation was performed using a greedy decoding as we are more interested in a small answer. 3.2.1. Generative question answering 3.2.3. Post-processing answer
The model generates and explains an answer for each During the system evaluation phase, a response is coninput text. In order to simplify the next step of aggre- sidered correct if it matches exactly one of the plausible gating diferent answers, it is essential to provide some ones. Therefore, this phase performs simple operations reasoning in addition to the answer during this phase. to clean up the response as much as possible: conversion After experimenting with various instructions, we identi- to lowercase, removal of punctuation and white spaces. ifed a good prompt for the generation model: ” A n s w e r t o t h e f o l l o w i n g q u e s t i o n i f i t i s p o s s i b l e , o t h e r w i s e r e t u r n N o n e . $ c o n t e x t : . . . $ q u e s t i o n : . . . G i v e t h e 4. Experiments r a t i o n a l e b e f o r e a n s w e r i n g ”
At each step, the $ c o n t e x t is replaced with text and the $ q u e s t i o n with the real question, resulting in a comprehensive input prompt for the language model. The generation was performed using a beam search decoding with 5 beams, a temperature setted to 0.4 and a fixed number of new tokes equals to 100 as we are interested in a meaningful sentence generation.
In this section, we report empirical evaluations of our open domain QA system. All the experiments reported concern only the Open domain QA system, and not the information retrieval system, which aims to generate a collection of textual documents from diverse sources, whether open or internal. As the resulting corpus varies depending on the specific use case (for internal purposes, internal documents supplemented with selected web news are preferred, whereas for general tasks, a Reading comprehension QA SQUADv2 TriviaQA rc web dev Open-domain generative QA TriviaQA unfiltered web dev TriviaQA rc web dev
Reading comprehension QA: The system is required to extract the correct answer from an average of six input documents, which exclusively consist of evidence documents provided by TriviaQA for each question. Although the input corpus is smaller, this task is more challenging than a standard reading comprehension task, as there are no designated passages and the answer could potentially be in any of the input documents. 4.2.1. Reading comprehension QA
the SQUADv2 and rc TriviaQA datasets since they are commonly used for this purpose. We used the passages Wikipedia dump alone may sufice), the input corpus of from the Squad dataset and the collection of evidence docdocuments was fixed to evaluate the OpenQA system. uments from TriviaQA for each question, so performance is mostly influenced by the question answering system. 4.1. Datasets The table 4.1 shows our results for both datasets. Our system does not achieve state-of-the-art results 10 on the We consider the following datasets. SQUADv2 dataset, but this is not our main objective. We
SQUADv2 [19] combines the 1,000,000 questions from are not aiming to excel on a specific dataset or machine SQUAD1.1 with approximately 50,000 unanswerable ques- reading comprehension task where the passage is already tions generated by crowd workers to resemble answer- provided, and the system must retrieve the answer. Conable questions. The SQUADv2 dataset, comprising a set sequently, models that are trained on SQUADv2 dataset of questions related to a text passage, is better suited for for such a specific task tend to perform much better. On a reading comprehension task where we need to extract the other hand, the results related to the TriviaQA rc web the answer from the passage. dev dataset are comparable to those found in the state
TriviaQA [20] is a reading comprehension dataset of the art 11, excluding those based on large language containing over 650K question-answer-evidence triples models (billions of parameters) as our model has 750M that were originally scraped from the Web. TriviaQA is parameters. Nevertheless, a more careful and specific more complicated than SQuAD since answers may not be evaluation should be carried out for the other existing immediately derived by span prediction and the context systems to assess whether they have been tested in the is sometimes extensive (on average, each document con- same scenario, especially in a zero-shot regime. tains 6.580 word pieces). The dataset 9 includes almost 487K documents, with 413K from the web domain and 4.2.2. Open-domain QA 74K from Wikipedia.
4.2. Evaluation ument corpus, consisting of approximately 487K documents. However, this fixed input corpus is not comparaIn order to evaluate the system, we used the evidence ble to open-domain QA systems that typically use larger documents provided by the TriviaQA dataset as our input corpora, such as Wikipedia dumps [4]. The performance corpus. We then measured the performance of the system, shown in the table 4.1 on the TriviaQA unfiltered web as described by Rajpurkar et al. [19], using two standard dev dataset is inferior to that of more modern open QA metrics. The exact match metric (EM), which measures systems 12. This is probably due to the fact that in this the percentage of predictions that match any one of the initial evaluation of the system, we considered a fixed ground truth answers, and the micro F1-Score (F1), which initial corpus, and the unfiltered TriviaQA dataset does evaluates the average overlap between the prediction not guarantee that the answer is present in the docuand ground truth answer. Our system was tested in two ments provided, so in many cases, the system may not be scenarios: able to find the answer. Finally, the table 4.1 also reports
Open-domain generative QA: The system must re- the performance on the TriviaQA rc web dev dataset for trieve the correct answers from a corpus of almost 487K this open domain scenario. These performances can be documents from, TriviaQA dataset, in response to each question. 10https://paperswithcode.com/sota/question-answering-on-squad20 11https://paperswithcode.com/sota/question-answering-on-triviaqa 12https://paperswithcode.com/sota/ open-domain-question-answering-on-kilt-2?metric=EM
compared with the previous ones in the table 4.1, and as expected, the F1-score decreases by 5%, as the retriever is more deceived by the large number of documents in the corpus.
In this paper, we investigate a method to develop an openQA system, which relies on retrieving documents containing information relevant to the query and extracting the answer using an instruction fine-tuned language model, following a “zero-shot” approach. Despite its conceptual simplicity, we show that combining a “zero-shot” and a prompt engineering strategy is a practical and promising approach to building a flexible open-domain question-answering system. In future work, we would like to compare our whole system, including information retrieval, to the state-of-the-art. [9] Y. Mao, P. He, X. Liu, Y. Shen, J. Gao, J. Han,
W. Chen, Rider: Reader-guided passage reranking for open-domain question answering, 2021. [10] T. B. Brown, B. Mann, N. Ryder, M. Subbiah,
J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, 2020. [11] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis,
H. Hajishirzi, L. Zettlemoyer, Rethinking the role of demonstrations: What makes in-context learning work?, 2022. [12] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay,
W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, [1] D. Chen, A. Fisch, J. Weston, A. Bordes, Reading J. Devlin, A. Roberts, D. Zhou, Q. V. Le, J. Wei, ScalWikipedia to answer open-domain questions, in: ing instruction-finetuned language models, 2022. Proceedings of the 55th Annual Meeting of the As- [13] D. Chen, A. Fisch, J. Weston, A. Bordes, Reading sociation for Computational Linguistics (Volume wikipedia to answer open-domain questions, CoRR 1: Long Papers), Association for Computational abs/1704.00051 (2017).
Linguistics, 2017, pp. 1870–1879. [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, [2] M. J. Seo, J. Lee, T. Kwiatkowski, A. P. Parikh, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, AtA. Farhadi, H. Hajishirzi, Real-time open-domain tention is all you need, 2017. question answering with dense-sparse phrase index, [15] K. Guu, K. Lee, Z. Tung, P. Pasupat, M.-W. Chang, CoRR abs/1906.05807 (2019). Realm: Retrieval-augmented language model pre[3] V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, training, 2020.
S. Edunov, D. Chen, W. tau Yih, Dense passage re- [16] A. Roberts, C. Rafel, N. Shazeer, How much knowltrieval for open-domain question answering, 2020. edge can you pack into the parameters of a language [4] G. Izacard, E. Grave, Leveraging passage retrieval model?, in: Proceedings of the 2020 Conference with generative models for open domain question on Empirical Methods in Natural Language Proanswering, in: Proceedings of the 16th Confer- cessing (EMNLP), Association for Computational ence of the European Chapter of the Association Linguistics, 2020, pp. 5418–5426. for Computational Linguistics: Main Volume, As- [17] N. Reimers, I. Gurevych, Sentence-bert: Sentence sociation for Computational Linguistics, 2021, pp. embeddings using siamese bert-networks, CoRR 874–880. abs/1908.10084 (2019). [5] M. Glass, G. Rossiello, M. F. M. Chowdhury, A. R. [18] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Naik, P. Cai, A. Gliozzo, Re2g: Retrieve, rerank, Pre-training of deep bidirectional transformers for generate, 2022. language understanding, 2019. [6] W. Yang, Y. Xie, A. Lin, X. Li, L. Tan, K. Xiong, M. Li, [19] P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, SQuAD: J. Lin, End-to-end open-domain question answering 100,000+ Questions for Machine Comprehension of with bertserini, CoRR abs/1902.01718 (2019). Text (2016). [7] S. Wang, M. Yu, X. Guo, Z. Wang, T. Klinger, [20] M. Joshi, E. Choi, D. Weld, L. Zettlemoyer, TriviW. Zhang, S. Chang, G. Tesauro, B. Zhou, J. Jiang, aQA: A large scale distantly supervised challenge R3: Reinforced reader-ranker for open-domain dataset for reading comprehension, in: Proceedquestion answering, CoRR abs/1709.00023 (2017). ings of the 55th Annual Meeting of the Association [8] P. Qi, H. Lee, O. T. Sido, C. D. Manning, Answering for Computational Linguistics (Volume 1: Long Paopen-domain questions of varying reasoning steps pers), Association for Computational Linguistics, from text, 2021. 2017, pp. 1601–1611.