The increasing volume of scientific literature and the abundance of publicly accessible data present a substantial hurdle for researchers aiming to stay informed and effectively derive valuable insights. In this paper we discuss the use of LLMs in the context of extracting information from scientific literature and introduce an AI-driven Research Assistant that uses custom Retrieval Augmented Generation (RAG) as a Service and other techniques to streamline processes such as literature review, information extraction, and knowledge discovery.
Large Language Models (LLMs) have gained significant attention across diverse industries, with their remarkable reasoning abilities enabling time savings and idea generation across numerous domains. Scientific research stands to benefit greatly, however, the probabilistic nature of LLM inference can lead to inaccurate responses for specialized queries. To address this, the novel paradigm of Retrieval Augmented Generation (RAG) has emerged over the last years. RAG enhances LLM query context by retrieving factual information from external sources such as documents, databases, and APIs, thus mitigating reliance solely on LLM training data [1].
Furthermore, the RAG paradigm not only improves the accuracy of LLM outputs but also expands their capabilities by enabling access to real-time information and specialized knowledge bases. This dynamic integration of external information allows LLMs to evolve beyond their static training data and stay abreast of the latest developments in rapidly changing fields such as scientific research.
While Retrieval Augmented Generation (RAG) offers significant improvements in LLM performance, it is not without limitations. The common RAG approach of retrieving top-k semantically similar passages based on vector embeddings and metrics like cosine similarity can encounter challenges. For instance, queries demanding multi-hop reasoning or complex relationships can fall short. Consider the query: "Which is the GDP of the country where the highest mountain peak in the world is?" While a RAG system trained on geography data may accurately answer "Which is the highest mountain peak in the world?", the second query necessitates identifying the country associated with the mountain and then retrieving its GDP, a task potentially beyond simple semantic similarity matching.
A variety of techniques are being explored to overcome these limitations [2]. These include:
• Reranking: Employing a two-step retrieval process, first retrieving a larger set of candidates using efficient methods like cosine similarity, then reranking them using more sophisticated techniques like cross-encoders or even LLMs themselves, to better capture relevance and address multi-hop reasoning. • Hybrid RAG: Combining semantic similarity-based retrieval with knowledge graphs to incorporate explicit relationships and facilitate more complex reasoning [3]. • Large Context LLMs such as Claude with its 200k token context or Gemini with its 1M+ token context, can significantly enhance RAG systems by incorporating more extensive context directly into the model. This reduces the need for explicit retrieval of external information in many cases, allowing the LLM to draw upon a broader knowledge base to understand and reason over larger chunks of information [4]. • Hierarchical Embeddings: Leveraging embeddings of the original text alongside various levels of summaries, such as in the RAPTOR model, to enhance retrieval accuracy and efficiency [5]. • Multi-Hop Query Answering: Decomposing complex queries into simpler sub-queries and employing techniques like Chain-of-Thought prompting to guide LLMs through multi-step reasoning [6,7,8]. • Autonomous Agents: Utilizing AI agents to navigate diverse information sources and construct more intricate prompts for the LLM, incorporating logic and multi-step reasoning [9]. • Self-RAG: Where the LLM itself participates in the retrieval process, potentially leading to more adaptive and context-aware retrieval [10].
These advancements highlight the active research and development in the RAG paradigm, aiming to address its limitations and enable LLMs to tackle increasingly complex and nuanced information needs.
Despite these advancements, constructing a RAG system specifically for scientific research remains a difficult task. Challenges arise in handling diverse content types commonly found in research papers, such as tables, images, and formulas, which often necessitate conversion into natural language text for effective retrieval. Moreover, the research process often involves snowballing, where the corpus of relevant references expands iteratively from an initial set of studies.
A review of existing AI assistants and RAG systems tailored towards researchers reveals a lack of a universally applicable solution, although several platforms demonstrated advanced techniques and some even appeared customized for scientific research. Existing solutions in this area can be categorized into three distinct classes:
• AI Document Assistants: Constituting the most prevalent category, these solutions range from freely available to premium licensed offerings. While they leverage cuttingedge LLMs and enable users to upload documents in various formats, answer questions based on those documents, and perform summarization, they often exhibit limitations specifically concerning academic research support. These constraints include operating on a limited number of documents (typically restricted by the LLM's context window), lacking specialization in scientific papers, and generally not performing multi-hop reasoning across multiple documents. [21], detailed information about its features and performance remains limited. However, given Clarivate's established track record, the product has strong potential to become a noteworthy contender in this space.
To overcome the above mentioned limitations we initiated the development of a RAG as a Service Research Assistant whose features we briefly introduce in the following section.
Researchers can upload their own PDF papers or initiate automated downloads for specific queries through the implementation of ArXiv and Semantic Scholar APIs, thus having access to millions of papers. GROBID processes papers, resulting in structured XML representations with clearly defined sections, figures, tables, and references [22]. The automated download function can also recursively expand the corpus by extracting references from the initial document set.
Full plain text sections from papers are indexed in ChromaDB [23] using cutting-edge BAAI/bge-m3 [24] dense embeddings. Deviating from most Q&A RAG systems, we employ large paragraph chunks (often entire chapters/sections) to maintain context. Oversized paragraphs are divided into subsections while preserving sentence integrity. Question answering employs cosine similarity retrieval, with results re-ranked using BAAI/bge-reranker-v2-m3 [25]. In cases where no relevant documents are retrieved, the system transparently informs the researcher that the answer is not grounded in the corpus. Each answer is accompanied by source paper sections, promoting transparency and facilitating further exploration.
Our AI Research Assistant facilitates the extraction of custom-structured information from scientific papers by generating valid JSON schemas from natural language queries. The standard schema extracts a comprehensive set of data including definitions, indicators, hypotheses, key findings, topics, and summaries from each paper. These summaries can be indexed within the vector database as well, enabling the system to also respond to high-level conceptual queries, in contrast to specific questions grounded in isolated paper sections which is addressed by the previously mentioned Q&A RAG features. Cost analysis indicates that leveraging a Model as a Service (for instance, Claude Haiku [26]) for information extraction incurs an estimated cost of 1 USD per 250 papers.
The system adopts a decoupled multi-server architecture for scalability, with LangChain [27] partially managing orchestration. A GPU-intensive machine is required for vector embedding, re-ranking, and local inference when utilizing local LLMs. The workflow exposes API endpoints that can be consumed by a web application to manage user access.
The project incorporates agentic behavior, utilizing a ZERO-SHOT Classifier to direct user queries to the most suitable tool. These tools include classic RAG/Q&A based on articles, structured information extraction, and deterministic queries to SQL-like datasets (e.g., locally hosted EUROSTAT data). LangChain's agentic implementation allows for chaining multiple tools in a single user query. Therefore, an answer to a user query can be grounded in both the scientific corpus and a relevant dataset.
Our approach achieves deterministic and explainable results through:
• Deterministic LLM use (temperature set to 0)
• Grounding question-answering in the scientific literature corpus and indicating the paper sources along the answer • Enforcing JSON schemas for structured information extraction • Verbose mode for agentic workflow, enhancing explainability • Integration of conventional programming tools for querying structured data sources This research assistant offers a significant contribution to AI-powered literature analysis, providing researchers with a valuable tool for navigating the expansive landscape of scientific knowledge.
We aim to integrate image and figure extraction from papers, leveraging multi-modal LLMs to enrich the dataset. Open models such as PaliGemma [28] or Idefics [29] can do image-to-text inference locally on reasonably accessible hardware.
Related to multi-hop reasoning, we plan to employ techniques like query decomposition and graph neural networks to address complex, multi-step queries.
Additionally, we plan to develop more agentic tools capable of handling various datasets, expanding the system's capabilities and adaptability.
In the mid to long term, we intend to enhance our system by integrating knowledge graph retrieval with the existing semantic similarity-based approach. This knowledge graph will be constructed during the paper processing and parsing stage, extracting relevant entities like topics, affiliations, and named entities as nodes. This hybrid approach aims to facilitate more nuanced and complex query handling, enabling the AI Research Assistant to better understand and leverage the intricate relationships within scientific literature.
This research is financed under the Romanian National Recovery and Resilience Plan, by the Romanian Government, under the contract number 268/29.11.2022, Entitled "CAUSEFINDER -CAUSALITY IN THE ERA OF BIG DATA".