The increasing volume of scientific literature and the abundance of publicly accessible data present a substantial hurdle for researchers aiming to stay informed and effectively derive valuable insights. In this paper we discuss the use of LLMs in the context of extracting information from scientific literature and introduce an AI-driven Research Assistant that uses custom Retrieval Augmented Generation (RAG) as a Service and other techniques to streamline processes such as literature review, information extraction, and knowledge discovery.
A variety of techniques are being explored to overcome these limitations [
Reranking: Employing a two-step retrieval process, first retrieving a larger set of candidates using efficient methods like cosine similarity, then reranking them using more sophisticated techniques like cross-encoders or even LLMs themselves, to better capture relevance and address multi-hop reasoning.
Hybrid RAG: Combining semantic similarity-based retrieval with knowledge graphs to
incorporate explicit relationships and facilitate more complex reasoning [
Large Context LLMs such as Claude with its 200k token context or Gemini with its 1M+
token context, can significantly enhance RAG systems by incorporating more extensive
context directly into the model. This reduces the need for explicit retrieval of external
information in many cases, allowing the LLM to draw upon a broader knowledge base to
understand and reason over larger chunks of information [
Hierarchical Embeddings: Leveraging embeddings of the original text alongside various
levels of summaries, such as in the RAPTOR model, to enhance retrieval accuracy and
efficiency [
Multi-Hop Query Answering: Decomposing complex queries into simpler sub-queries
and employing techniques like Chain-of-Thought prompting to guide LLMs through
multi-step reasoning [
Autonomous Agents: Utilizing AI agents to navigate diverse information sources and
construct more intricate prompts for the LLM, incorporating logic and multi-step
reasoning [
Self-RAG: Where the LLM itself participates in the retrieval process, potentially leading
to more adaptive and context-aware retrieval [
These advancements highlight the active research and development in the RAG paradigm, aiming to address its limitations and enable LLMs to tackle increasingly complex and nuanced information needs.
Despite these advancements, constructing a RAG system specifically for scientific research remains a difficult task. Challenges arise in handling diverse content types commonly found in research papers, such as tables, images, and formulas, which often necessitate conversion into natural language text for effective retrieval. Moreover, the research process often involves snowballing, where the corpus of relevant references expands iteratively from an initial set of studies.
A review of existing AI assistants and RAG systems tailored towards researchers reveals a
lack of a universally applicable solution, although several platforms demonstrated advanced
techniques and some even appeared customized for scientific research. Existing solutions in this
area can be categorized into three distinct classes:
• AI Document Assistants: Constituting the most prevalent category, these solutions
range from freely available to premium licensed offerings. While they leverage
cuttingedge LLMs and enable users to upload documents in various formats, answer questions
based on those documents, and perform summarization, they often exhibit limitations
specifically concerning academic research support. These constraints include operating
on a limited number of documents (typically restricted by the LLM's context window),
lacking specialization in scientific papers, and generally not performing multi-hop
reasoning across multiple documents.
• General purpose RAG as a Service: Augmenting document assistants with retrieval
capabilities enables access to a substantially larger corpus of documents. Solutions
within this category encompass both commercial and freely available options. The
following are noteworthy examples:
•
• RAGflow: An open-source RAG engine with a comprehensive feature set and a
user-friendly, modern interface [
To overcome the above mentioned limitations we initiated the development of a RAG as a Service Research Assistant whose features we briefly introduce in the following section.
Researchers can upload their own PDF papers or initiate automated downloads for specific queries through the implementation of ArXiv and Semantic Scholar APIs, thus having access to millions of papers. GROBID processes papers, resulting in structured XML representations with clearly defined sections, figures, tables, and references [22]. The automated download function can also recursively expand the corpus by extracting references from the initial document set.
Full plain text sections from papers are indexed in ChromaDB [23] using cutting-edge BAAI/bgem3 [24] dense embeddings. Deviating from most Q&A RAG systems, we employ large paragraph chunks (often entire chapters/sections) to maintain context. Oversized paragraphs are divided into subsections while preserving sentence integrity. Question answering employs cosine similarity retrieval, with results re-ranked using BAAI/bge-reranker-v2-m3 [25]. In cases where no relevant documents are retrieved, the system transparently informs the researcher that the answer is not grounded in the corpus. Each answer is accompanied by source paper sections, promoting transparency and facilitating further exploration.
Our AI Research Assistant facilitates the extraction of custom-structured information from scientific papers by generating valid JSON schemas from natural language queries. The standard schema extracts a comprehensive set of data including definitions, indicators, hypotheses, key findings, topics, and summaries from each paper. These summaries can be indexed within the vector database as well, enabling the system to also respond to high-level conceptual queries, in contrast to specific questions grounded in isolated paper sections which is addressed by the previously mentioned Q&A RAG features. Cost analysis indicates that leveraging a Model as a Service (for instance, Claude Haiku [26]) for information extraction incurs an estimated cost of 1 USD per 250 papers.
The system adopts a decoupled multi-server architecture for scalability, with LangChain [27] partially managing orchestration. A GPU-intensive machine is required for vector embedding, re-ranking, and local inference when utilizing local LLMs. The workflow exposes API endpoints that can be consumed by a web application to manage user access.
The project incorporates agentic behavior, utilizing a ZERO-SHOT Classifier to direct user queries to the most suitable tool. These tools include classic RAG/Q&A based on articles, structured information extraction, and deterministic queries to SQL-like datasets (e.g., locally hosted EUROSTAT data). LangChain's agentic implementation allows for chaining multiple tools in a single user query. Therefore, an answer to a user query can be grounded in both the scientific corpus and a relevant dataset.
Our approach achieves deterministic and explainable results through: • • • • •
Deterministic LLM use (temperature set to 0) Grounding question-answering in the scientific literature corpus and indicating the paper sources along the answer Enforcing JSON schemas for structured information extraction Verbose mode for agentic workflow, enhancing explainability
Integration of conventional programming tools for querying structured data sources This research assistant offers a significant contribution to AI-powered literature analysis, providing researchers with a valuable tool for navigating the expansive landscape of scientific knowledge.
We aim to integrate image and figure extraction from papers, leveraging multi-modal LLMs to enrich the dataset. Open models such as PaliGemma [28] or Idefics [29] can do image-to-text inference locally on reasonably accessible hardware.
Related to multi-hop reasoning, we plan to employ techniques like query decomposition and graph neural networks to address complex, multi-step queries.
Additionally, we plan to develop more agentic tools capable of handling various datasets, expanding the system's capabilities and adaptability.
In the mid to long term, we intend to enhance our system by integrating knowledge graph retrieval with the existing semantic similarity-based approach. This knowledge graph will be constructed during the paper processing and parsing stage, extracting relevant entities like topics, affiliations, and named entities as nodes. This hybrid approach aims to facilitate more nuanced and complex query handling, enabling the AI Research Assistant to better understand and leverage the intricate relationships within scientific literature.
This research is financed under the Romanian National Recovery and Resilience Plan, by the Romanian Government, under the contract number 268/29.11.2022, Entitled “CAUSEFINDER CAUSALITY IN THE ERA OF BIG DATA”. [16] https://github.com/Mintplex-Labs/anything-llm [17] https://github.com/kanad13/RAGify [18] C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha, The AI Scientist: Towards Fully
Automated Open-Ended Scientific Discovery (2024). doi: 10.48550/arXiv.2408.06292 [19] https://insilico.com/science42/dora [20] https://www.unriddle.ai [21] G. Ben-Porat, Introducing the Clarivate Academic AI Platform, 2024. URL: https://clarivate.com/blog/introducing-the-clarivate-academic-ai-platform [22] GROBID [Computer software] (2008–2024). URL: https://github.com/kermitt2/grobid [23] https://www.trychroma.com [24] https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/BGE_M3 [25] https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/llm_reranke r [26] https://www.anthropic.com/news/claude-3-haiku [27] https://www.langchain.com [28] https://huggingface.co/google/paligemma-3b-pt-896 [29] https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3