A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics demand a robust memory system grounded in reality. In this paper, we propose such a memory system consisting of three components. First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception. Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to eficiently manage relational information. Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation. We illustrate the system's working and potential using a real-world example.
The emergence of Large Language Models (LLMs) has advanced conversational assistants beyond
rule-based systems, enabling them to operate within a user’s perceptual and conceptual context. For
this, assistants must integrate stored knowledge with ongoing interactions to ensure that responses
remain relevant and grounded. Retrieval Augmented Generation (RAG) techniques combine LLMs
with external knowledge bases and multimodal LLMs process diverse inputs, enabling richer, more
context-aware interactions. However, the need for assistants with personal and situational support
based on a large-scale memory also highlights critical challenges. First, to efectively deal with memories
in the form of multimodal inputs, a robust conceptual framework is needed that acknowledges the
role of space and time as fundamental dimensions of experience and memory. Inspired by Kantian
notions, which describe space and time as fundamental structuring elements imposed by the mind [
By combining these elements, our system enables assistants to deliver personalized, context-aware support with enhanced reasoning and decision-making.
CEUR
ceur-ws.org
This section reviews related work for the three pillars of our memory system: grounded perception,
memory graphs, and agentic retrieval. Grounded perception organizes multimodal data into actions,
agents, and objects with temporal awareness, forming structured action patterns [
Multimodal perception has advanced with LLMs, aiding AI applications like robotics and surveillance.
For instance, robots critically depend on their visual understanding capabilities for navigation [
A scalable memory is essential for assistants with personal support capabilities. Due to their benefits
regarding the integration of heterogeneous data [
GraphRAG is a retrieval-augmented generation that enriches conventional RAG pipelines with a
graphbased representation of knowledge. In standard RAG systems, semantic search is used to retrieve
relevant text snippets from a vector store, which are provided as context to an LLM for question
answering. However, this chunk-oriented retrieval can miss deeper relationships and dependencies
among pieces of information. GraphRAG addresses this limitation by building and utilizing knowledge
structured in graphs, enabling more coherent reasoning over interconnected facts. One way of realizing
GraphRAG is to conduct semantic search to find entry points in the graph and then expand the context
for further relevant, but less explicit, information. There are several suitable algorithms for graph
expansion, PageRank being one of them [
The memory system is based on a schema that revolves around textual notes, which are represented as nodes in a graph, cp. Section 3.1. Leveraging this schema, the memory system is designed to seamlessly capture, structure, and retrieve real-world observations through a three-phase process, see Figure 1.
In the perception phase, cp. Section 3.2, cameras observe the environment, allowing a VLM to generate descriptive captions for detected events. While this can be extended for further modalities such as audio, we focus on visual inputs in the context of this paper. During the ingestion phase, cp. Section 3.3, these images and captions undergo a structured analysis before being stored in a persistent knowledge graph. Unlike unstructured memory systems, this graph-based representation explicitly encodes who performed which action on which object, when, and where. Finally, in the recall phase, cp. Section 3.4, the system retrieves stored information for question answering, event verification, and intelligent recommendations. Throughout the following, we rely on a video showing an individual in a home setting as a running example to exemplify the concepts introduced.
The knowledge base builds on a schema for representing so-called memory notes, cp. Figure 2. A MemoryNote can be used to describe a time period of arbitrary length and it can be generated from arbitrary sources, e.g., manually crafted for diary entries or generated automatically to describe a single frame in a video. Each memory note is characterized by its note content, which is a natural language string, and an optional list of data files from which it has been created. To create a structured representation, every memory note is also represented as a node in a knowledge graph. For our application, we introduce Image nodes, which are specialized memory notes that have an image caption as note content and that refer to an image as a data file. To create a structured representation, each MemoryNote is linked to the entities it mentions, categorized as Agents (”Who performed the action?”), Objects (”What was acted upon?”), and Actions (”What was done?”). Images are temporally ordered using has-previous links, cp. Figure 2, and agents, objects, and actions are connected to the images they occur in via has-element links.
The perception phase captures raw video input images, and generates descriptive captions using gpt-4o’s [21] vision capabilities, thus laying the foundation for a structured representation of events.
Unlike traditional video memory systems that passively store raw visual inputs, our system actively identifies and links key entities in the environment. This includes detecting agents, objects, and their spatial relationships, forming a structured representation. To achieve this, the system processes sequences of consecutive frames from the video stream using the prompt shown in Figure 3. Each sequence is analyzed as a single unit, where the first and last frame overlap with adjacent sequences, ensuring temporal continuity. To find a balance between eficiency and accuracy, we caption only each n-th frame. This strategy maintains temporal coherence, reduces redundant descriptions, and results in more accurate, context-aware scene summaries. Each described instance in the captions is indexed with a unique label in the format [label_x:Type] to ensure consistent tracking across frames.
The information captured during the perception phase, cp. Section 3.2, is stored in a hybrid knowledge base combining a knowledge graph consisting of structured relationships and a vector store, i.e., a text-oriented representation allowing for semantic search. The ingestion process consists of four steps. First, all entities identified during the perception phase are extracted from the image captions. Together with the entity names, we extract the entity types. Second, we create embedding vectors for the image captions using an embedding model, resulting in high-dimensional numeric representations of the text. If necessary due to context window limitations, the captions are split up into several parts. Third, we create nodes in the knowledge graph for all images and connect them sequentially. To these we add the respective attributes, such as the captions and the paths for the image files, and the embedding vectors for the image captions. Fourth, we add nodes for all actions, agents, and objects identified and connect them to all the image nodes in which they appear, turning the sequence of images into a connected graph. The knowledge graph, cp. Figure 4, maintains temporal order via sequentially connected image nodes, while objects, agents, and actions structure events. Consistent entity labels ensure continuity and enable context-aware retrieval.
Combining a graph-based structured representation with natural language text notes allows the use of several retrieval techniques. First, we use the memory system as a standard RAG system for semantic search as it is based on natural language notes with embedding vectors associated to them. By embedding the user query with the same embedding model, we create an embedding vector which we compare to the embeddings of all the notes, retrieving the semantically most relevant notes. To increase the relevance of the context provided to the LLM, we optionally use a reranker to check the retrieved notes and filter out less relevant ones. Providing the retrieved notes as context, we let the LLM answer the original question. This type of retrieval is especially eficient for questions which are likely to have responses that are semantically close. Second, we leverage the structure of the memory system for graph expansion. Relevant information may be included in notes which are not found when relying purely on semantic search, but that are linked to the notes found in the graph. This is usually the case for relevant background information, e.g., personal preferences not showing up in individual notes, but represented in a note for an agent. Here, we start with regular semantic search for identifying an initial set of relevant notes. Then we expand the search results using an expansion algorithm. Specifically, we use PageRank [22], but other algorithms, e.g., random walks, also work. Finally, the expanded search results are used as context for LLM-based question answering. This second type of retrieval is beneficial whenever there is implicit background information, which cannot easily be found via semantic search, but which becomes apparent when analyzing the surrounding environment of the relevant nodes in the graph. Third, we rely on the graph representation for structured information retrieval in the form of text2cypher. For this, we let the LLM translate the user input into a Cypher query, which is run against the graph database. The result in the form of a table is interpreted by the LLM, which formulates a natural language response. This type of retrieval is ideal for answering structure-oriented queries, for instance questions that require counting entities. To leverage the benefits of all three retrieval techniques, we combine them in an agentic retrieval system. For this, we wrap the three retrieval functionalities into tools that we provide to an LLM-based agent, who can access them as needed for answering user questions. The agent is prompted to select the most suitable tool, or several if necessary, to retrieve information from the memory system and eventually formulate an appropriate answer.
We implemented a prototype using Neo4j as the graph database and vector store. Videos were recorded using an iPhone 15, split into frames at 3 Hz, and captions were generated for every fith frame using gpt-4o for vision and text-embedding-3-small for storing embeddings in the vector store. New entities (images, actions, agents, objects) are inserted into the graph using dynamically generated Cypher queries. Memory notes are also stored in Obsidian, enabling easy browsing. A minimalist gradio frontend allows users to interact with the system via a chat interface, cp. Figure 5, that provides links to the sources used for transparency. Table 1 presents representative questions and system responses, demonstrating retrieval capabilities across diferent query types.
This paper presents a grounded memory system that integrates the strengths of a knowledge graph and a vector store for agentic RAG with an LLM as an intuitive natural language interface The system leverages a minimalist schema and operates through three key phases: perception, ingestion, and retrieval. The system has potential applications ranging from robotics to assistive technologies, such as support systems for dementia patients. Our approach provides a foundation for structured memory-based retrieval and serves as a starting point for future research in long-term knowledge representation and context-aware reasoning. The integration of conceptual nodes provides additional flexibility, allowing retrieval to be guided by semantic relationships rather than purely temporal order. This structured approach enables conversational assistants to reason over past events, improving long-term memory consistency compared to standard RAG techniques.
Future work should focus on scaling up the system and conducting large-scale evaluations in realworld scenarios. Expanding to longer multimodal sequences will allow the system to capture broader temporal dependencies and leverage its efectiveness in retrieving and reasoning over complex event histories. While we expect challenges in long-term entity disambiguation – ensuring that agents, objects, and actions are consistently recognized across diferent scenes and timeframes – moving beyond individual observations enables high-level behavior pattern identification by recognizing repetitive actions, activity trends, and structured sequences of human interactions. Additionally, we will further advance RAG techniques, such as recursive summarization and query rewriting, to enhance contextual understanding and improve response accuracy. This can be supported by advancing agency in the retriever and providing further retrieval tools, e.g., via frameworks focusing on leveraging large sets of tools for LLMs [23]. Further improvements will also involve extending the system’s multimodal capabilities beyond vision, incorporating audio and spatial information. In doing so, we aim to move toward a comprehensive memory system capable of supporting autonomous agents in complex environments.
During the preparation of this work, the authors used generative AI for grammar and spelling checks. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [21] J. Achiam, et al., Gpt-4 technical report, arXiv:2303.08774 (2024). [22] S. Bin, K. L. Page, The anatomy of a large-scale hypertextual web search engine, in: Computer
Networks, 1998. [23] F. Ocker, D. Tanneberg, J. Eggert, M. Gienger, Tulip agent–enabling LLM-based agents to solve tasks using large tool libraries, arXiv preprint arXiv:2407.21778 (2024).