The widespread availability of search API's (both free and commercial) brings the promise of increased coverage and quality of search results for metasearch engines, while decreasing the maintenance costs of the crawling and indexing infrastructures. However, merging strategies frequently comprise complex pipelines that require careful tuning, which is often overlooked in the literature. In this work, we describe NeuralSearchX, a metasearch engine based on a multi-purpose large reranking model to merge results and highlight sentences. Due to the homogeneity of our architecture, we could focus our optimization eforts on a single component. We compare our system with Microsoft's Biomedical Search and show that our design choices led to a much cost-efective system with competitive QPS while having close to state-of-the-art results on a wide range of public benchmarks. Human evaluation on two domain-specific tasks shows that our retrieval system outperformed Google API by a large margin in terms of nDCG@10 scores. By describing our architecture and implementation in detail, we hope that the community will build on our design choices. The system is available at https://neuralsearchx.neuralmind.ai.
Metasearch engines provide a unified interface for searching and aggregating results from diferent sources. They take advantage of existing search engines to increase the diversity of results while decreasing maintenance costs for crawling and indexing infrastructures.
This approach, however, comes with a unique set of challenges: many times metasearch engines choose to consult only sources that might be relevant to a given query, since searching all available collections may be unfeasible due to resource constraints, such as limited bandwidth. For example, given a medical-related query, a metasearch engine could choose to consult only search engines that are capable of returning medical-related content. Therefore, a metasearch engine must represent the capabilities and type of data that each source search engine provides, i.e., the representation problem. When receiving a query, the metasearch engine must be able to select the appropriate sources, i.e., the selection problem. Furthermore, a metasearch engine is also responsible for generating a unified list of results from the multiples lists that it retrieves from, i.e., the merging problem.
In this work, we introduce NeuralSearchX, a multi-stage metasearch engine that takes a
diferent approach to metasearch: to improve the quality of results we retrieve mainly from
well-established general-purpose engines, such as Google and Bing, as well as curated sparse
and dense indexes. Therefore, we avoid the representation and selection problem as we can
simply search all our bases for any given query. To address the merging problem, NeuralSearchX
leverages state-of-the-art Transformer [
Our approach does not require any specific knowledge from the source collections since it ranks documents purely by content. We show in the experimental section that our system outperforms well-established search engines even without extra information from the document or its source. NeuralSearchX also takes advantage of low-reliability cloud infrastructure in its deployment in order to use high computing power while keeping a low overall cost.
Our contributions are the following: • We propose zero-shot Transformer rerankers as an efective merging strategy for metasearch engines. We demonstrate the efectiveness of our reranking pipeline on various public datasets as well as human evaluations; • By using low-reliability infrastructure, we show that it is possible to use a model with billions of parameters in a production environment under a reasonable budget, thus allowing us to deploy a highly efective search engine;
In this section, we first discuss metasearch engines and their main challenges. Then we provide an overview of multistage retrieval pipelines.
Metasearch engines typically support hundreds or even thousands of search engines and
therefore demand highly elaborated solutions for the representation, selection and merging
problems [
Metasearch engines traditionally handle the collection representation problem by maintaining
representational sets that contain pertinent information for each searchable collection. Such
sets can be created manually [
For the selection problem, existing solutions often calculate similarity metrics between the
query and the representational sets of each collection [
Traditional merging strategies are based on scoring functions that take into account textual
content and metadata retrieved from the source search engines. For example, CORI [
A multistage ranking retrieval pipeline is a common retrieval pipeline that uses multiple
consecutive refinement steps to return the final results. This method was first referenced
back in 2006 at Matveeva et al. [
In the last decade, we have seen an increasing dedication to improving the quality of the
results by proposing innovations on diferent stages of the retrieval pipeline [
NeuralSearchX is a multi-stage metasearch engine: the first stage consists of a candidate document retrieval step in which documents are retrieved using federated search over web search engines along with sparse and dense retrieval from private collections. The second stage consists of a reranking step of the previously selected documents with a neural model. The model scores each document by how relevant it is for a given query. The best scored documents are then sent to a highlighter model that estimates and selects the most relevant sentences from each document. Finally the documents are formatted and returned to the end user. Figure 1 illustrates the described pipeline.
For the first stage, we use both sparse and dense retrieval methods provided by the Pyserini’s [
Before indexing, we split the contents of long documents over multiple windows. We do this by splitting the original content in words and grouping them according to a certain window size and stride. Typically we use windows of 150 words with a stride of 75 words. We use 150 words because it translates to about 250 tokens, which is close to the max length of 256 tokens used to train our models. This process is done to improve the eficiency of the Transformer model used in the next stages, since it has a quadratic cost over the length of the input sequence. It also improves its efectiveness since this window length is close to the average length of the texts that the model was finetuned on, i.e., the passages from the MS MARCO dataset.
Furthermore, NeuralSearchX also leverages third-party APIs, such as Bing, Google and Semantic Scholar, to retrieve documents from the web. We assume that the snippets returned by such APIs are representative enough of the original page; therefore we use the provided snippets as the document content. A single search can use multiple of such sources to create a diverse set of documents. Since all the retrievals are independent, we can run them in parallel, therefore not creating a significant overhead in the process and maintaining an adequate latency.
After the candidate document retrieval, the next step is to merge all the candidates in a single
list, and then rank such documents so that the most relevant ones are on the top of the results
list. To do that, NeuralSearchX uses one of the following rerankers:
• A mMiniLM reranker introduced by Bonifacio et al. [31]. It has 107 million parameters
and was trained on the English and Portuguese subsets of the mMARCO [31] dataset, a
machine translated version of the MS MARCO dataset [32]. This distilled model has been
shown to perform better than models with an order of magnitude more parameters [
The reranker computes the relevancy of each document for a given query. After all documents are scored, the list of results is reordered according to these scores.
With the results ordered by their relevance, NeuralSearchX takes the first ten ranked documents and performs a highlighting step, whose goal is to take long documents and select the most important sentences to show the user, therefore reducing the amount of text that the user needs to read. Note, however, that this step is not performed when the document text is already a snippet (such as those from Google and Bing), because the text in this case is already of an appropriate length. We begin by splitting the document into sentences and compute the relevance of each sentence to the query. Since the task is still the same as in the previous stage (only the text is shorter), we can leverage the same model and hardware used in the ranking stage. For each document, the two sentences with the highest scores are selected and then shown in the user interface.
Like most modern search engines, NeuralSearchX was built in a cloud environment to take advantage of its distributed computing capabilities. We made sure however that our system was cloud agnostic, meaning it is easily adaptable to any major cloud provider. At the time of writing this paper, we had already deployed the system on Google Cloud and Microsoft Azure.
NeuralSearchX uses a frontend in a Single-page Application (SPA) model, and is built with React, a well-established javascript library for the creation of web pages with dynamic content. The webpage is served through an Nginx web server that runs on a dedicated machine. The same Nginx web server also acts as a load balancer for the requests to the API.
Our API is built using FastAPI, a Python framework for the construction of APIs based on the Starlette toolkit. FastAPI is capable of creating very well-performing and low-latency APIs. The API is responsible for handling document retrieval using both Pyserini’s retrieval methods and through http requests to third-party APIs. It also handles basic parsing and formatting of documents.
In recent years, we have seen increasing improvements delivered by neural models for information retrieval, by far surpassing traditional search methods with respect to the quality of results [34]. However, such models come with a major drawback: they are much more computationally expensive than the traditional non-neural search.
Due to this high computational cost, such models either have a high latency on modest hardware or demand costly hardware (mainly GPUs) to perform computations in reasonable time, making such methods less attractive for commercial search engines.
In the last couple of years, we also saw a common trend among the major cloud providers in the form of spot resources. Such an infrastructure is cheap but can be taken down by the provider with little to no warning. The use of low-reliability infrastructure was explored for training huge models [35, 36, 37] but little work has been published on its use in production systems, especially those that demand high computing power, such as systems powered by deep learning models. NeuralSearchX takes advantage of spot infrastructure to make the deployment of state-of-the-art models at a feasible price, allowing for both low query latency and a competitive low-cost infrastructure. Our work shows a practical use case that spot instances can be used with success in a deployment scenario.
Furthermore, in our experience, the spot resources that we use (mainly machines with T4 gpus) are not evicted often: in a period of three months, our available hardware was below 80% of our total capacity only once. However, the time to start serving a model in a new machine can be up to 20 minutes, accounting for both the time to create a new container and loading the necessary models onto the GPU.
The major drawback of using spot hardware is the lack of reliability, since resources can be deallocated at any given time. To mitigate this problem, we deploy our spot resources under a Kubernetes cluster that handles the allocation of spot machines. If a machine is taken down, Kubernetes promptly starts the process to allocate a new one with the equivalent services.
Being prepared to recover from resource eviction is essential when using spot machines. Depending on the scale and characteristics of the system in question, additional mechanisms may be necessary. For example, a hybrid scheme using both spot and non-spot resources would provide a reliable minimum computation while still achieving an adequate average capacity.
The most expensive part of our pipeline is the reranking and highlighting stages as they use modern GPUs to do hundreds of inference passes on a large neural network in under a few seconds. Such resources are deployed on a low-reliability infrastructure in a Kubernetes cluster. Currently we use a managed cluster in Azure Kubernetes Service (AKS), but this could be replaced with any other managed Kubernetes service from the major cloud providers. All neural models are served using TorchServe, a performant and reliant solution for the deployment of Pytorch [38] models. Each model has its own TorchServe image, which is deployed in pods that run on nodes of the Kubernetes cluster. This method of provisioning the reranker service allows us to modify and adapt the system in a seamless way. For example, we were able to deploy a model with 3.7 billion parameters by creating a torchserve image with the respective model and then deploying it under 10 machines with T4 GPUs. The API would simply separate the passages to be scored in 10 requests, which will then be distributed between the 10 nodes running the model service and return the results of the inference.
Two fundamental factors that determine the usefulness of a retrieval system are query latency and quality of results. In this section we analyze both of these aspects in NeuralSearchX, as well as its serving cost.
To evaluate the eficiency of our system we measure the queries per second (QPS) that our
system can handle and compare it with its monthly cost. We also compare our results with
those presented by Wang et al. [
We use Locust,1 a Python framework for load testing, to measure the system QPS. We evaluate the eficiency of our system in the MS MARCO dataset with a standard BM25 for document retrieval. We also evaluate our search in the general web using third-party APIs. The simulated users from Locust were configured with the following characteristics: 1. Each simulated user sends a random query from the query set. 2. After each request, the simulated user waits 1 to 15 seconds to issue another query.
Before the load test, we make sure that all relevant components are deployed under the same private network and were already deployed under the Kubernetes cluster, so we evaluate the system in its peak eficiency.
In the MS MARCO load test, we used the MS MARCO passage dev topics as our query set. The system was tested for 15 minutes. For the Web load test, we used a custom query set containing English and Portuguese queries. For this test the system was tested for 3 minutes. The results of the load tests for both models are shown in Table 2. The column “90%’ refers to the query latency of the 90th percentile.
The projected monthly cost of our system is shown in Table 1. In comparison to MS Biomedical Search Engine, we were able to achieve a similar QPS but with a much lower overall cost. The main reason for this is the ability to use spot instances for GPU-intensive services, as discussed previously. Business
pt
MRR@10 nDCG@3
To evaluate search systems, numerous document collections were created and made publicly available. To measure the efectiveness of NeuralSearchX, we indexed two of such document collections and their respective automatic translations to Portuguese: 1. MS MARCO [32]: A large scale dataset introduced in 2016 comprising more than 8 million passages and 500 thousand queries issued to the Bing search engine and their respective relevant passages selected by humans. Our models were trained in query and document pairs from this dataset. 2. Robust04 [39]: A benchmark that contains half million documents from the news domain and a large amount of annotations for 250 queries.
As mentioned previously, NeuralSearchX is capable of answering queries in languages other than English, since it uses multilingual models to rank the retrieved documents. To evaluate our system in other languages, we used the translated versions of MS MARCO and Robust [31]. We tested using the Portuguese version of such datasets as this is the language spoken by the majority of our users. Unless otherwise noted, the experiments on MS MARCO and Robust04 uses 1000 documents from the candidate retrieval stage.
The results are shown in Table 3. Both models used in NeuralSearchX outperform BM25 by a large margin in multiple metrics and datasets. The mT5-3B reranker shows the best results for all metrics and datasets except for the MARCO dataset due to the fewer candidate documents provided to the model in that experiment. Note that while the mMiniLM model shows lower numbers than mT5-3B, it has much better eficiency, making it an attractive option to reduce costs.
To validate the quality of our results from web sources, we developed an interface in which annotators see two lists for each searched query. One of the lists displays only the Google API results in the same order that they were provided by Google, while the other list displays the results in the order given by NeuralSearchX. This list contains both Google and Bing documents. To hide the identity of the search engine, we swap the positions of the two lists with 50% probability each time a query is issued. The top 10 results in each list are labeled by simply clicking on a button on the interface, which is shown in Figure 2.
We then contracted attorneys to create 50 queries and annotate their results when they were issued to our system and Google Search. All queries were in Portuguese and related to Brazilian law. For each query, the attorneys labeled the first 10 documents from our search engine and the first 10 returned by Google.
Using these labels, we computed the metrics presented in column “Legal” of Table 3. NeuralSearchX outperformed Google API by more than 8 nDCG@10 points. This experiment shows that our method can surpass well-established search engines in certain domains and languages, by reranking the results obtained from third-party APIs.
Our system was selected in November 2021 to power the search engine of SEBRAE, a Brazilian entity that supports micro and small enterprises. Our system outperformed four other competitors and was chosen mainly by the quality of its results in a human evaluation.
SEBRAE provides a couple of databases for search. One of them contains long technical responses to questions from small businesses and entrepreneurs. SEBRAE receives such questions and reaches out to experts in the subject, who in turn create a technical document that answers the question in detail. Another database is from an open forum called "Sebrae Respostas" where entrepreneurs can ask anything. It works as a normal forum, but SEBRAE staf tend to answer most questions with relevant information.
We extracted the text from the provided databases to construct a common sparse index. In addition to the databases provided, we also use Bing and Google search APIs to increase the diversity of results. We leave it to our reranker to decide which documents are shown to the end user. This system is available at quest.neuralmind.ai.
To evaluate how well our pipeline handled the documents from the business domain, we used the "technical responses" previously mentioned. Each technical response had the original question that originated the document annotated. We collected all such questions and used them as queries to evaluate the system. For each of such queries, the only document that was considered relevant was the one from which the question was extracted.
Our first result on this dataset showed a very high MRR for both NeuralSearhX and BM25. To make the tests more discriminatory, we reduced the query set to those that performed poorly in the previous experiment by either search method. We ended up with approximately 600 test queries. We show the results for our more discriminatory test in Table 3, column “Business + MRR@10”.
Moreover, we performed a manual graded relevance annotation, i.e., the documents are marked by humans as non-relevant, on topic or relevant, on a curated set of 25 queries provided by SEBRAE. The annotation was performed on the top 3 results from the Google API, BM25 and NeuralSearchX. The results are shown in Table 3, column “Business + nDCG@3”. NeuralSearchX clearly outperforms both BM25 and the Google API by a large margin.
In this work we presented NeuralSearchX, a metasearch engine that takes advantage of the latest advancements in pretrained deep learning models as well as low-reliability infrastructure to make the system afordable. We demonstrated its eficiency through numerous experiments, even when using billion-parameter machine learning models. In particular, we demonstrated competitive QPS with MS Biomedical Search while having a much lower overall cost. Finally, we showed the efectiveness of NeuralSearchX in popular public datasets and general web search, and demonstrated that it can outperform strong commercial search engines such as Google. Furthermore, we described our experience in solving and evaluating a particular use case in the business domain, showing the potential of our system for searching private collections.
This research was partially funded by grants 2020/09753-5 and 2022/01640-2 from Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP). We also would like to thank Microsoft Azure and Google Cloud for credits to support this work. arXiv preprint arXiv:2102.10073 (2021). [31] L. H. Bonifacio, I. Campiotti, R. de Alencar Lotufo, R. Nogueira, mMARCO: A multilingual version of MS MARCO passage ranking dataset, CoRR abs/2108.13897 (2021). URL: https: //arxiv.org/abs/2108.13897. arXiv:2108.13897. [32] P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al., MS Marco: A human generated machine reading comprehension dataset, arXiv preprint arXiv:1611.09268 (2016). [33] R. Nogueira, Z. Jiang, R. Pradeep, J. Lin, Document ranking with a pretrained sequence-tosequence model, in: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 708–718. [34] J. Lin, R. Nogueira, A. Yates, Pretrained transformers for text ranking: Bert and beyond,
Synthesis Lectures on Human Language Technologies 14 (2021) 1–325. [35] L. Bottou, F. E. Curtis, J. Nocedal, Optimization methods for large-scale machine learning,
Siam Review 60 (2018) 223–311. [36] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, et al., Large scale distributed deep networks, Advances in neural information processing systems 25 (2012). [37] S. Yang, S. Khuller, S. Choudhary, S. Mitra, K. Mahadik, Scheduling ml training on unreliable spot instances, in: Proceedings of the 14th IEEE/ACM International Conference on Utility and Cloud Computing Companion, 2021, pp. 1–8. [38] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, Advances in neural information processing systems 32 (2019). [39] E. M. Voorhees, et al., Overview of the TREC 2003 robust retrieval track., in: Trec, 2003, pp. 69–77.