Digitalizing construction site workflows requires handling heterogeneous data from various sources in a userfriendly way. Linked Building Data (LBD) ofers a powerful framework for this but remains largely inaccessible to on-site users unfamiliar with Semantic Web technologies. To address this, we developed a natural language interface integrated into a 3D construction site-focused LBD viewer, enabling intuitive access to structured and unstructured data without requiring SPARQL or RDF expertise. Our system employs a locally hosted large language model (LLM) and a Retrieval-Augmented Generation (RAG) pipeline to retrieve and generate responses based on both static and real-time data. We detail the architecture, implementation, and trade-ofs of three RAG design approaches and present a robust, prebuilt query pipeline tailored for dynamic construction site use cases. This work demonstrates the feasibility, limitations, and potential of LLM-enhanced access to LBD for non-expert users in real-world construction environments.
The construction industry is characterized by a multitude of heterogeneous data sources, creating a
fragmented digital landscape. Interoperability across these sources is essential for efective
collaboration and addressing environmental, social, and economic sustainability goals—especially as
construction increasingly adopts digital and robotic technologies. Linked Building Data (LBD) ofers a
promising approach to improve interoperability [
This is mainly due to two factors: (1) on-site users predominantly lack familiarity with Semantic Web
technologies like SPARQL and RDF, and (2) most existing tools require technical setups, e.g. bringing
a laptop to the work area, which is deemed unsuitable for field environments. This gap has been noted
across academic and industry settings, and got particularly evident in the EConoM research project,
which investigates AI, edge computing, and 5G for constructions site management. One of its key
use cases, autonomous intralogistics, relies on integration and user interaction of data about logistics
processes, building elements, materials, and real-time telemetry from mobile robots [
To improve accessibility for such use cases, we began developing a web-based software prototype.
As a foundation, we adapted the open-source three.js-based FOG demo application by Mathias Bonduel
[
ROS), access to non-RDF sources like time-series databases, and reduced reliance on SPARQL.
Motivated by these needs and inspired by the LDAC Summer School 2024 and its ”LLM+RAG+Ontologies”
repository2, we explored a Retrieval-Augmented Generation (RAG) approach powered by a local large
language model (LLM) [
This paper presents the design and implementation of an RAG-powered LBD interface tailored for real-world construction environments. We evaluate diferent RAG strategies and propose a robust architecture based on prebuilt queries, assessing its feasibility, limitations, and potential for broader application.
Not only since the wide availability of LLMs and RAG techniques, semantic data processing with the
help of natural language has sparked interest among researchers exploring new ways to retrieve,
analyze, and interact with construction data. In 2016, Lin et al. [
Zheng et al. [
LLM models face challenges such as inaccurate responses and made-up answers. Uhm et al. [
The primary use case for the system was its deployment on the local edge server of the reference construction site in Aachen, a full-scale construction testbed at RWTH Aachen University. This computational core, running a large number of dockerized services, is connected to a dedicated 5G campus network, enabling low-latency, high-throughput communication with all machines and devices on-site.
Due to the network topology and security constraints, the RAG system had to be split into at least two
components: a user interface (UI) running on the user’s device, and backend RAG services deployed
on the edge server, where access to databases and machine APIs is provided. Several existing solutions
could be integrated into the LBD viewer with minimal modification for the UI. Since the edge server
is equipped with four Nvidia A40 GPUs (48 GB VRAM each), deploying the LLM locally via Ollama
[
However, multiple approaches to implementing RAG exist, each with distinct trade-ofs. The following section outlines three options we investigated, along with a comparative summary of their respective strengths and limitations (see Table 1).
Vector Database with Data Embeddings This method involves storing data as dense embeddings in a vector database to capture semantic relationships. Embeddings are fixed-size, high-dimensional vectors generated by the LLM to represent concepts or text. Similarity search over these vectors enables retrieving the most relevant content without explicit keyword matching. While powerful for static, document-like data, this approach performs poorly for dynamic or real-time use cases due to the need for frequent, costly re-embedding.
Fully LLM-generated Queries This approach leverages the LLM’s ability to dynamically create SPARQL or SQL queries based on natural language input and an understanding of the data schema. For example, the model can convert a user query into a valid SPARQL command by referencing an embedded ontology. While this enables flexible and expressive querying, the generated queries often sufer from syntax errors or semantic mismatches, especially in heterogeneous data environments. Pipeline with Prebuilt Queries and APIs This method employs a modular execution pipeline in which the LLM selects and parameterizes prebuilt or partially prebuilt scripts. These are connected to specific RDF stores, time-series databases, and real-time data streams. Although this approach requires more manual setup, it ensures predictable behavior, simplifies testing and debugging, and allows for ifne-grained control over sensitive or safety-critical operations.
Given the dynamic nature of construction site processes, machine data, and the architectural constraints of the edge server infrastructure, the pipeline approach with prebuilt queries emerged as the most suitable option. Unlike other methods, it supports persistent RDF datasets and near-real-time data streams or machine control via dedicated APIs. This flexibility allows the system to bridge static knowledge bases and dynamic sources while maintaining architectural modularity. A major strength of this approach is its predictability: prebuilt queries and scripted logic ensure stable and repeatable behaviour. This is particularly important when user interactions do not simply retrieve data but can also create new data entries or trigger machine actions. In such contexts, reliability and system safety take precedence over flexibility. Although the pipeline approach requires more upfront development efort, we considered this trade-of essential to ensure robust performance and operational security. In contrast, while theoretically more flexible, dynamically generated queries based on embedded ontologies proved error-prone, dificult to debug, and hard to predict in production scenarios. Repetition of similar queries (e.g., highlighting specific building elements) also introduces unnecessary computational overhead without added user value.
To implement a functional prototype, we designed a layered system architecture using modular, opensource tools, specifically from Open WebUI’s pipelines toolchain 4. The architecture follows a clear separation of concerns: each layer is responsible for a specific function within the data retrieval and interaction pipeline. This modularity improves maintainability, allows distributed deployment across constrained edge hardware, and ensures safe execution of potentially sensitive operations such as machine control or data creation.
The architecture consists of five layers (see Figure 2):
• User Interface Layer: Enables user interaction through natural language queries and visualization of data. Built by embedding Open WebUI’s chat interface into a Three.js-based LBD viewer. • Instruction Allocation Layer: Matches user intent with available functions and API calls. Implemented using custom Python code adapted from Open WebUI’s pipelines toolkit. • Data Retrieval Layer: Fetches information from RDF stores (via SPARQL), time-series databases, and real-time sources like MQTT or ROS. • Context Augmentation Layer: Enriches the user query with retrieved data to prepare a structured prompt for the LLM. • Response Generation Layer: Uses a locally deployed LLM to generate responses. This layer can also interact with the viewer via MQTT to update tables or visual highlights.
The existing JavaScript implementation was updated to use a newer version of the Comunica library for more advanced SPARQL queries. Due to changes in the library’s API, this required modifying the internal logic used to process and display result sets. We chose React/Node.js as the frontend framework to improve modularity and maintainability, replacing global JavaScript variables with React Context, Hooks, and Props. This change also enabled the integration of React Three Fiber for 3D rendering.
The conversational chat interface is based on Open WebUI and is embedded into the application via an HTML iframe. The backend LLM agent communicates with the frontend through MQTT channels, pushing updates such as SPARQL query results or structured messages. These are used to populate tables, highlight 3D elements, or provide feedback directly in the chat window (see Figure 3). While the modular design allows for easy extension and reconfiguration, it also introduces complexity in user session handling and authentication. For example, ensuring consistent identity across the LBD viewer and the embedded chat remains a challenge and is the subject of ongoing work.
One of the first and simplest functionalities to test the system was retrieving LBD data by querying
for specific building elements, using filters based on Building Element Ontology (BEO) classes or the
Building Topology Ontology (BOT). For the BEO, this was achieved by passing the variable Etype
(element type) into the Python script, which was then used in a simple SPARQL SELECT query. Since
BEO aligns closely with IFC class names, mapping the user’s query to the nearest IFC class and passing
it as Etype worked well for most elements. When using language-agnostic models (we tested Llama
3.1 (8B and 70B) [
To ensure functions could be discovered and invoked by the Instruction Allocation layer, they have to be clearly described. This includes what each script does, what parameters it takes, and what it returns (see Listing 1).
Listing 1: Description for the python definition retrieving the OLCC transport details. 1 """ 2 Gets the detail information of a OLCC transport for a specific transport ID. 3 4 :param id: The ID of the specific OLCC transport for which we want more information. 5 :return: The detailed information about the specific OLCC process as markdown. 6 """
This description was used for a function retrieving logistics process data from the Online Logistics Control Center (OLCC), used by Zeppelin Rental GmbH for tracking and billing of material transports. In our observed use cases, users typically start by requesting all inbound shipments for a given day. A follow-up query, such as “Return all data for OLCC ID 187005,” triggers the function to fetch the shipment details. The SPARQL query used in the backend is shown in Listing 2.
Listing 2: Query for the logistics data which belong to a certain OLCC ID. 1 PREFIX olcc: <http://w3id.org/olcc#> 2 PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> 3 4 DESCRIBE ?resource 5 WHERE {?resource olcc:buchungsId "{id}"^^xsd:integer.}
The resulting RDF graph is pre-processed and returned in multiple ways. To keep the chat interface clean, only a short message is displayed, while full tabular data is transmitted via MQTT to the viewer and shown in the application table. An interesting feature of the response generation layer is that it can distinguish between internal “knowledge” (e.g., JSON results not shown to the user) and final userfacing output. This allows, for example, internal logistics metadata like olcc:ladezone (loading zone) to remain in context and be retrieved later when the user asks follow-up questions such as “Where is the last known position of the package?”
More complex queries require more attention on the data retrieval layer, such as prompting for
scheduled construction progress, as shown in Figure 3. For our construction site data model, we can
query planned and accomplished construction progress using the Internet of Construction Process
Ontology (IoC)5, specifically the concepts of ioc:Schedule and ioc:Status. However, the instances we want to
iflter are modelled using xsd:dateTime literals according to ISO 8601 [
To address this, we implemented a lightweight solution using an instruction-tuned language model (Mistral-Nemo-Instruct) to convert natural language date expressions into ISO 8601 format. The function can be accessed directly by the instruction allocation layer or triggered as a fallback when a regex check on a date parameter fails for ISO compliance. To ensure reliable parsing, we made eforts to prevent the model from adding annotations or altering the structure of the output. While instruct models are fine-tuned to follow explicit prompts, they also retain general world knowledge. For example, prompting the model with “the day Columbus reached the new world” correctly produces the ISO-compliant literal ”1492-10-12T02:00:00Z” for use in filtering queries.
Data availability can certainly be improved with the presented RAG approach using LBD. More
promising, however, are the opportunities to generate data on demand in the field, e.g., for the planning and
execution of robotic processes, as there are currently no accessible methods for doing so [
The prompt contains the parameters of what to transport (”Paket1875-1”), where to transport it to 5https://internet-of-construction.github.io/IoC-Process-Ontology/ (”WorkStorage_5”), and which machine to use (”Innok223”). At the current stage, these parameters must either be explicitly provided by the user or inferred from the context of previous answers. While this approach enables flexibility, it also poses challenges to usability, as users may not always know or remember the exact identifiers needed. We consider this a limitation of the current implementation and plan to integrate more robust disambiguation and auto-suggestion mechanisms in future iterations. The chat response shows that one process has been instantiated and also mentions that six child processes have been created. These processes depend on the machine requested. The Innok Robotics Heros 223 is an autonomous ground vehicle (AGV) equipped with an automatic trailer coupling system. This means that logistics processes with this AGV follow a reference process scheme that is first queried. It consists of loading the trailer with the requested payload (1), moving the AGV to the trailer position (2), coupling the trailer (3), transporting to the target position (4), decoupling (5), and moving the AGV out of the target zone (6).
Generating the process sequence involves several validation steps. If any check fails—such as missing material location or unavailable target zone—the system responds with clarification requests or error messages. We plan to extend this validation further to include capability matching and integration with the machine scheduling system. Once the processes are instantiated, the backend services handle path planning and execution, which can be launched immediately if the schedule permits. As seen in Figure 4, the resulting path planning and current operation status are both formalized in RDF and visualized in the 3D viewer. While the system currently operates with predefined process templates, the underlying architecture could be extended to support standardized workflow definitions, such as BPMN, and executed through workflow engines (e.g., Camunda, WSO2). However, introducing such complexity would require additional semantic modeling efort and more granular user control over process instantiation and lifecycle management.
As the RAG pipeline has access to a Linked Data model, including machines, their network endpoints, associated APIs, and the 5G network infrastructure, it can be extended to retrieve machine data or even trigger control mechanisms on the construction site with relative ease. The web app can be used, for example, on a tablet and operated via voice using the integrated microphone and speech-to-text interface. Command success rates depend on the strictness settings of the pipeline, which can be adjusted by setting the model’s temperature, and on the clarity of the user’s speech. Figure 5 shows examples of how reality is shadowed in the Digital Twin.
For the autonomous transportation robot, the system can access onboard sensors and retrieve media
streams, such as images or video, from its cameras. Through a time-series database, users can also
query machine telemetry, such as battery status (shown at the bottom of Figure 5). In addition, live
streams of data like rotational speed, location, or the robot’s point cloud, used for obstacle detection,
can be made available. These point cloud updates (visualized as red dots) provide temporal, real-time
information not typically modeled. For example, the red line at the lower center of Figure 5 represents
the open door of a construction container. Based on previous work in project [
In addition to data retrieval, the system can also be used for actuation—controlling devices through exposed APIs. A practical example is a construction lift that can move vertically. Here, voice commands ofer tangible benefits, allowing workers to control the lift remotely without dismounting from a vehicle. While this control functionality is technically feasible, it introduces critical safety concerns. Unlike passive data queries (e.g., requesting an onboard image), misinterpreted or imprecise control commands could lead to unsafe actions such as activating the wrong equipment. For this reason, safeguards must be put in place to ensure that potentially hazardous control operations are only executed with clear user intent and proper authentication.
The presented RAG-based system for accessing Linked Building Data (LBD) and other construction site information demonstrates both significant strengths and critical limitations. To evaluate its overall impact, we outline the main strengths, challenges, opportunities, and associated risks.
A key advantage of the system is its ability to lower the entry barrier to LBD and other complex, structured data environments for non-expert users on construction sites. Users unfamiliar with SPARQL, RDF, or system-specific APIs can now interact with LBD models and digital twins using natural language. This enhances usability and accessibility compared to traditional interfaces. Beyond data retrieval, the system enables advanced functionalities such as initiating autonomous robotic processes or accessing real-time sensor streams without requiring technical expertise. The conversational interface contributes to user confidence by guiding interactions, requesting missing parameters, and providing feedback instead of cryptic errors.
While the use of prebuilt queries ensures predictable and safe behavior, it also limits system flexibility. The current approach restricts interactions to predefined templates and cannot yet accommodate arbitrary queries or dynamically composed logic. Error handling remains a challenge, particularly when users input vague or malformed parameters. This is especially noticeable in speech-based interaction, where identifiers like WorkStorage_5 are prone to misinterpretation. As the system scales to include more data types and interaction patterns, retrieval accuracy and validation become increasingly important to ensure reliable operation and user trust.
Future improvements should address scalability and adaptability. They could integrate specialized LLMs optimized for structured retrieval, improving accuracy while reducing manual setup. Agentbased frameworks capable of refining user prompts, selecting appropriate data sources, and validating retrieved content could further increase robustness. The ability to instantiate and monitor robotic processes opens pathways toward adaptive scheduling, predictive diagnostics, and seamless humanmachine collaboration. These extensions would support real-time decision-making and broaden the accessibility of robotic workflows in construction.
Integrating AI-based interaction into operational construction systems introduces serious safety and security concerns. While the current implementation enforces strict limitations on executable actions, the potential for voice or text-based commands to trigger real-world machine operations necessitates rigorous safeguards. Misinterpretations, unauthorized access, or malicious prompts could result in hazardous scenarios. Future iterations must implement multi-layer authentication, role-based permissions, and validation checkpoints before executing any critical command.
This work presents an approach for improving access to Linked Building Data through a
RetrievalAugmented Generation (RAG) architecture powered by a locally hosted large language model (LLM).
The system enables natural language interaction with complex structured and real-time data, lowering
the barrier to entry for non-expert users in construction. By embedding a conversational interface into
a 3D viewer and integrating prebuilt queries, the system supports both robust information retrieval and
the generation of autonomous workflows, such as robotic material transport. While the pipeline
approach ensures stability and safety, challenges remain regarding query flexibility, input robustness, and
system scalability. Future work will focus on improving query interpretation, expanding support for
dynamic instruction allocation, and refining safety mechanisms for interacting with physical systems.
Recent developments in agent-based architectures [
This work is part of the EConoM research project funded by the Federal Ministry for Digital and Transport of Germany within the initiative InnoNT (funding number 19Ol22009F). It was supported within the TARGET-X framework, a project funded by the Smart Networks and Services Joint Undertaking (SNS JU) under Horizon Europe (funding number 101096614). The authors are responsible for the content.
During the preparation of this work, the authors used a local chatbot based on Mistral Nemo 12b and Deepl for spelling checks, translations and paraphrasing for shortening the text. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.