The complexity of modern construction projects necessitates collaboration among diverse stakeholders and the handling of substantial data volumes. Significant investments in digitization have occurred globally to address this complexity, emphasising the need for interoperability among standards and diverse knowledge sources. The emergence of Digital Building Twins (DBTs) further underscores the importance of integrating heterogeneous data to create comprehensive digital representations of buildings. DBTs enable real-time data integration and support various phases of architectural development, ofering practitioners access to historical, present, and predictive data. In this context, our research focuses on enhancing the accessibility and interpretability of DBT data through natural language querying. Leveraging domain-specific ontology and advanced AI techniques, our approach facilitates eficient communication between users and DBTs, enabling rapid extraction of specific building details. This paper presents our methodology, including knowledge representation, semantic analysis, and information extraction, along with evaluation results demonstrating its efectiveness in improving DBT querying performance.
Construction projects are complex and require collaboration between various stakeholders to
develop increasingly intelligent structures. Smart constructions entail handling substantial data
volumes, which must be organised for services like digital archives, data analysis, and predictive
interpretation. Significant investments in digitisation occur globally to address the complexity
of information required for decision-making in today’s construction industry, necessitating
interoperability among the underlying standards used. Historically, GIS (Geographic Information
Systems) addresses the modelling of the environment surrounding a construction, while BIM
(Building Information Modelling) tackles the construction itself. To have a complete model of
constructions and their environment, one must integrate GIS and BIM standards, thus blurring
traditional distinctions between the two domains, highlighting the need to remove barriers to
information sharing across disciplines, ultimately optimising investments and enhancing data
quality [
Additionally, Digital Building Twins (DBT) represent a crucial aspect of modern construction
processes. A DBT entails real-time communication among sensors to create a comprehensive
digital representation of a building at a specific moment, reflecting the evolving understanding
of construction in the digital era. Typically relying on Building Information Modelling (BIM), a
DBT represents the virtual counterpart of a physical structure, replicating its real attributes and
conditions [
To best understand the challenges associated with accessing information from a DBT, let us consider the example of a large building with a surface of 10,000 square meters and 50 stories, which is a structure that typically has numerous emergency exits. Querying information about these exits, including their dimensions and compliance status, involves using some software tool to navigate through diferent levels and inspect potential existing door labels such as "fire exit" or "emergency exit." This task is time-consuming, taking at least a dozen minutes for experienced users and up to 30 minutes for less seasoned practitioners, particularly if thoroughness is required. Additionally, obtaining dimensions of door frames often requires guesswork, as this information may not be readily available.
Our approach aims to eficiently address such queries by providing specific building details
that may not be directly represented in the existing standard schema of the DBT. The approach
emphasises the efectiveness and user-centric design of the enhanced information retrieval
system presented in [
Following our definition of a DBT, this article presents our approach for an intuitive "dialogue" between humans and DBTs by rendering the data within a DBT accessible and interpretable via machine processing. Our approach allows expert and novice users to ask queries in natural language by leveraging domain-specific and semantically rich ontology crafted by domain experts, i.e. the OB27AI ontology. As such, it leverages advanced capabilities derived from the domain of Artificial Intelligence (AI), namely symbolic AI (through knowledge representation), deep learning and Natural Language Processing (NLP).
In this paper, we focus on improving the performance of queries addressed in natural language
over knowledge representations of DBTs. Our contributions are summarised as follows:
• We describe a complete approach for natural language querying of DBTs
• We define a new knowledge representation of DBTs in the form of the OB27AI ontology
and detail its design and contents
• We enhance our approach presented in [
The article is structured as follows: section 2 provides some definitions forming the necessary scientific background for our research, and section 3 resumes existing research done in the related domains. Our approach is introduced in section 4, while its implementation is provided in section 5. Last but not least, section 6 gives the results of the evaluation of our prototype, and section 7 concludes this paper.
The following sections provide the necessary definitions and references to understand our approach’s components.
Semantic Web (SW) aims to make heterogeneous information accessible to a machine, allowing
to turn data into machine-processable knowledge. SW uses an ensemble of W3C standards
for annotating data present on the Web and enabling knowledge graphs (KG). Based on the
existing Web standards, e.g. URI (Universal Resource Identifier) for resource identification and
HTTP as a universal access protocol, SW standards build upon RDF (Resource Description
Framework) for representing simple facts about resources (identified by HTTP URIs) and upon
the OWL (Web Ontology Language) family for describing more complex knowledge in the form
of ontologies. In an ontology, knowledge is expressed as "Subject - Predicate - Object" triples
and can be queried or modified using SPARQL (SPARQL Protocol and RDF Query Language).
Additionally, Linked Data (LD) has been defined by the W3C 1 as four principles for enhancing
the share and reuse of such annotated data to form a global network of knowledge accessible
and comprehensible by a network of machines. This link between diferent silos of knowledge
can be expressed in diferent ways, depending on the need. If it is necessary to align knowledge
for inference purposes (e.g. owl:equivalentClass), the reasoning mechanics provided by OWL
are to be preferred but can be costly; if the link to be established is strictly semantic, using the
vocabulary of an existing ontology such as SKOS is suficient [
The OWL family comprises several profiles for ontology description languages based on Description logic (DL) with diferent levels of expressivity. They allow for defining complex knowledge representations, combining concepts, relationships among concepts and constraints over these elements, generally expressed as logical rules or necessary and suficient conditions over class definitions. Given their symbolic and logical nature, it is possible to reason upon such knowledge representations by applying specific algorithms (called reasoners) that allow implicit axioms from the explicit ones contained in the ontology to be inferred. KGs are obtained when using LD principles to connect elements from diferent ontologies.
Building Information Modelling (BIM) has become vital in the last decade in AEC (Architecture,
Engineering and Construction). Following ISO 23386:2020 [
Industry Foundation Classes (IFC) [
OpenBIM also defines the exchange requirements for every type of business process and
every BIM design stage through Information Delivery Manuals (IDMs) [
As seen in section 2.2, the AEC industry has wide heterogeneity in existing knowledge and
data. The diversity and variation in information types, structures, formats, and semantics pose
interoperability challenges, as defined by ISO/IEC 2382:2015 [
Current standards outline three primary levels of interoperability: data, syntactic, and
semantic. These levels are interconnected and progressively enhance each other, with lower tiers
furnishing the necessary components for the functionalities of higher levels [
Data and syntactic interoperability are already extensively addressed by standard protocols (e.g. TCP/IP, HTTP) and the adoption of syntax standards (e.g. XML, HTML). Semantic interoperability, though, is yet to be fully addressed by existing standards and approaches.
Although BIM is considered state-of-the-art in the AEC industry, its adoption rate remains
relatively low [
NLP is a branch of AI that enables computers to understand, interpret, and generate human
language meaningfully. NLP utilises techniques from various fields, including linguistics,
computer science, and machine learning, to bridge the gap between human communication
and computational systems. Contrary to the formal knowledge models provided by ontologies,
language models are ambiguous and vague, while the relation between symbols to objects is not
clearly defined as in DL. Language models are defined as a "probability distribution describing
the likelihood of any string" [
The Transformer architecture, outlined in [
Built on the Transformer architecture, several approaches called Pretrained Language Models
(PLMs) have been published, i.e. BERT (Bidirectional Encoder Representations from
Transformers) or GPT (Generative Pretrained Transformer). As an extension of the Transformer
architecture, PLMs are a form of transfer learning that uses large and general text datasets to
train an initial NLP model with self-supervised learning objectives. The initial NLP model is
refined using smaller domain-specific text datasets [
There are three broad categories of PLM transformers: • Encoder-only (e.g. Bidirectional Encoder Representations from Transformers aka BERT) are meant to generate contextual representations of text. We aim to test SpaCy LLM, based on OpenLLaMA3 which uses Large Language Model Meta AI (LLaMA), developed by Meta. • Decoder-only (e.g. Generative Pre-trained Transformer aka GPT) are purposed to generate new text based on a context. We plan to test Falcon4, developed by TII (Technology Innovation Institute) UAE and GPT, developed by OpenAI, which is only available onpremises in version 25 (now in version 4). • Encoder-decoder (e.g. Text-to-Text Transfer Transformer aka T5) are geared toward transforming text. We seek to test Flan-T56, developed by Google, a T5 fine-tuned using instruction-based prompts.
NLP can be split into two main domains [
This section summarises existing standards and works in the literature about our approach’s diferent components.
To address the limitations of the IFC standard as mentioned in section 2.2, bSI organisation
promotes the ifcOWL ontology, which serves as a foundational domain ontology for the AEC
industry [
ifcOWL provides an OWL2-DL semantic representation of the IFC schema, which can be
used not only to access information in an IFC model, but also to link it with external data not
explicitly described in the IFC schema. Due to the verbose structure underlying the IFC data
model (more than 900 classes), the resulting OWL structure has a high OWL2-DL expressivity
(SROIQ(D)) [
When considering the use of ifcOWL for handling and interpreting DBT data, several types
of research have highlighted the need for less expressive structures [
“Ontology alignment is the process of identifying correspondences between semantically related
entities of diferent ontologies” [
As we have already seen, there is a wide disparity in the data around the field of AEC
construction engineering. Generally speaking, ontologies answer how to structure all this data,
enabling it to be expressed as knowledge. Moreover, depending on the formalism used (RDF
/ OWL / etc.), they enable us to define a balance between the expressiveness and decidability
of the knowledge they describe [
Aligning 2 or more ontologies to map the entities (ABox / TBox) of each domain under study
addresses the problem of semantic heterogeneity in the data [
There are 4 main approaches to ontological alignment [
They consist of comparing labels qualifying knowledge based on the notions of synonymy,
polysemy, translation or according to methods for calculating frequencies of occurrence
(TF-IDF, bags of words, etc.).
2. Approaches based on internal ontological structures (metadata, cardinalities, constraints,
attributes, etc.) and external structures (taxonomies) [
Ontology matching approaches are evaluated by the OAEI (Ontology Alignment Evaluation Initiative)11, which brings together all the research on the subject. Ontology alignment remains a real challenge for the scientific community.
Knowledge Base Question Answering (KBQA) systems aim to find answers to queries expressed
in natural language from a Knowledge Base (KB). Present methodologies for KBQA can be
8https://github.com/pipauwel/IFCtoRDF
9https://github.com/jyrkioraskari/IFCtoLBD
10https://github.com/kyriakos-katsigarakis/openmetrics/tree/master/openmetrics-kgg
11http://oaei.ontologymatching.org/
broadly categorised into two main groups:
1. Semantic Parsing Based Methods: These approaches focus on a semantic parser that
translates questions into intermediate logic forms. These logical forms are then executed
against the KB to retrieve relevant answers [
Recently, there has been a notable surge in research interest towards combining these two
approaches for KBQA [
When considering KBQA systems for querying DBT data, the semantic parsing of natural
language queries involves identifying IFC concepts, their relationships, and value restrictions.
As the existing ifcOWL ontology does not provide natural language descriptions or labels
for its elements (i.e. IFC concepts), the IFC Natural Language Expression (INLE) ontology
[
As seen in section 2.1, SW promotes using standardised vocabularies, ontologies and linked
data principles to enable machine-readable and interoperable information. Still, semantic
heterogeneity is persistent as knowledge representation is done through independent ontologies
with no existing links. Ontologies being formal knowledge models, one can consider standard
ISO 11354-1:2011 [
Integration harmonises elements/systems into a cohesive common whole, ensuring they conform to a common format for seamless interaction. It prioritises interoperability from the outset, ideal for new system designs requiring adherence to a common form. Specifically regarding ontologies, integration involves defining a new ontology and defining outgoing links to the elements of existing ontologies.
Unification establishes a shared meta-model for semantic equivalence, aiding translation
between diferent models. The meta-model ranges from a foundational reference vocabulary
to an intricate ontology. It maps all alternative models to this common framework, enabling
translation but potentially leading to some loss of information. Studies show it enhances
interoperability among AEC stakeholders [
Federation is decentralised, allowing entities to maintain autonomy without conforming
to a common format. Interoperability is achieved through mappings and agreements despite
diferences in vocabularies or methodologies. Unlike integration and unification, it doesn’t
require a common format or meta-model, allowing each entity to retain autonomy. This
approach suits contexts with diverse or intricate vocabularies and methodologies. As such, it has
higher design and implementation costs and provides the greatest flexibility. FOWLA (Federated
Architecture for OWL Ontologies) is one of the few examples of an ontology federation approach
[
We aim to enhance the general performance of the approach presented in [
The main modifications consist of a new domain ontology OB27AI (detailed in section 4.2) along with a dependency tree providing a syntactic understanding of the input sentence through constituency parsing coupled with an NL-enhanced model for grounding query elements (detailed in section 4.3).
To do so, we decided to develop a new domain ontology that would complement it, with input from domain experts, to have a finer-grain description of the desired subdomains while keeping the ontology simple enough for existing reasoners to work efectively.
There are many methods to develop a new domain ontology (e.g. TOVE, Seven-Step, SENSUS):
among these methods, the Seven-Step [
This methodology is iterative, and often, steps are revisited as the development process evolves. Each step builds on the previous ones, contributing to a comprehensive and robust ontology that accurately represents the knowledge in a particular domain. Additionally, continuous validation with domain experts and potential ontology users is essential throughout these steps to ensure its accuracy and usability.
The following paragraphs detail the methodology for conceiving our domain ontology (OB27AI). According to the Seven-Step methodology, we first determine our ontology’s domain and scope. We chose to focus on the following subdomains to encompass the DBT and its wider context and surroundings: • Structural works: the major works of a building (e.g. foundations, load-bearing walls and beams). • Heating, Ventilating and Air Conditioning (HVAC): all the fluids (e.g. water, air, gas) within a building. • Highways and miscellaneous external works: outside of a building (e.g. access and pathways, roads and parking areas, drainage networks, tanks), and the outdoor structural works (e.g. preparation of the site for foundations, fittings, excavation) • Electrical works: requirements and layout of electrical equipment, high-voltage (e.g. electrical power, lighting), low-voltage (e.g. telephone network, computer network), and ifre safety (e.g. detectors, fire alarms) • Materials: e.g. concrete, steel, aluminium, wood, cement. • Woodwork: similar to structural works but specific to wood-based structural elements. • All trade works: covers all of the above, as well as all the "second works" of a building (e.g. partitions, non-load-bearing walls, doors and windows, paint, skirting boards). For each domain above, domain experts enumerate the important terms and structure them into a taxonomy built top-down from a single high-level concept. They follow simple recommendations about naming conventions and guidelines regarding the expected structure. A group review of these taxonomies is then done to help the domain experts finalise them, to ensure the level of detail is in line with our objectives, and to check there are no duplicates of concepts across domains. The taxonomies are then processed automatically to fill the ontology. A manual review is done afterwards to check the consistency of the ontology, adapting it when needed while not deviating from the taxonomies. As outlined in the Seven-Step methodology, the whole process is iterative.
The TBox represents the entities and their relationships on a conceptual level, as shown in Fig. 2. The classes and the hierarchies we choose are similar to those found in the IFC ontology (version 2x3 TC1), though the following diferences and additions drive our work: • More classes are defined to materialise components of larger systems (e.g. door frame) for finer granularity. • The properties are simpler and straighter to keep the ontology more human-readable. • The natural language description of objects is prominent in the "b27:Name" class and its sub-classes, as it’s a key point to access the model and interact with the DBT.
The "b27:Name" is the root for all natural language representations of the concepts we chose
to focus on in our domain ontology, providing synonyms, hyponyms, hypernyms, abbreviations
and morphological variations, along with a similarity index related to the main (i.e. origin) term
representing a concept. As we aim to go deeper with a finer grain in our chosen subdomains
than the INLE ontology, we rely on this ontology (and its alignments) for a natural language
expression of the broader IFC concepts that may not be covered by our ontology. This approach
has been chosen to ease the interoperability with ifcOWL [
Listing 1: Example of a "b27:Name" subclass
Following the Seven-Step methodology, we identified the diferent ontologies we could reuse. ifcOWL is the most comprehensive ontology for IFC, which, in our approach, is complemented by INLE for handling natural language. As we create a new ontology, we maintain interoperability with ifcOWL and INLE through unification for a faster and tighter alignment. To do so, we embed the shared meta-model in our OB27AI ontology. A future venue would be a federation approach for interoperability, allowing our ontology to be fully independent, transposing the existing meta-model into mappings and agreements, and moving it out of our ontology.
We choose to proceed with alignment based on the internal structures of ifcOWL and INLE, which allows greater control over the inferences in the context of global reasoning. Our work is easier because INLE has been built as an extension of ifcOWL. Therefore, they are both already aligned. The considered ontologies have very few individuals in their original state, i.e., they are not linked to a model. Therefore, we focus on the schema of these ontologies, i.e., their classes and properties.
Our methodology has been inspired by Euzenat et al. [
• Semantic: considers the meaning of terms by relying primarily on domain experts. 3. Generate the meta-model to be embedded within our OB27AI ontology for most mapping based on constraints, equivalence, and subsumption. A more complex mapping part requires establishing inference rules outside our OB27AI ontology. 4. Validate the alignment using reasoning tools for inconsistencies and logical errors. A manual review is done with the domain experts to finalise the validation. 5. Refine the alignment each time a new taxonomy is incorporated into our OB27AI ontology by repeating this process on the extended ontology. 6. Integrate the alignments to produce our OB27AI ontology aligned with ifcOWL, and a modified INLE ontology aligned with ours (Fig. 3).
As briefly seen on Fig. 3, the INLE ontology has some generic individuals (e.g. "inle:ifcdoor1"). Our OB27AI ontology has such instances for properties, predefined values of properties, materials, etc., that are shown with more details in Fig. 4
However, most individuals are related to the names (and their variants, i.e. synonym, hyponym, lemma, etc.) in natural languages used by the domain experts, including common terms. Those are used to attach labels to the objects (e.g. door panel, door leaf), their properties (e.g. door aperture), the values of properties (e.g. automatic or manual door operation), etc.
As illustrated in Fig. 5, based on the labelling done in a building model (e.g. "Ridge beam"), the precise type of object (e.g. RidgeBeam) from our ontology is retrieved for an individual that is otherwise typed as a broader IFC class (e.g. IfcBeam). This process allows us to add constraints of all kinds (e.g. material, dimension, position, property, etc.) on these individuals, enabling us to reach our goal for a fine-grained description within the subdomains we chose.
We provide an example snippet another example of alignment around "b27:Door" in Listing 2.
Listing 2: Example of an OB27AI alignment The complete list of alignments can be found as a link set on the GitHub - donner le lien.
The complete definition of these alignments represents ongoing work and goes beyond the scope of this article. Besides corresponding to a correct application of Linked Data principles, such alignments provide the advantage of inheriting from properties and constraints defined in the other ontologies, thus allowing us to further reason on DBT knowledge. For example, aligning our classes with the BOT ontology allows us to take advantage of the diferent topological properties defined in BOT (Building Topology Ontology) ref - https://w3c-lbd-cg.github.io/bot/, thus allowing us to decompose the DBT into diferent spaces. Additionally, such alignments allow other ontologies to reuse the natural language annotation of building elements, as permitted by the B27AI ontology.
In our previous approach [
To improve our KBQA system, diferent approaches for the NER phase needed to be tested. The parsing of the input sentence is done on the semantic level using a convolutional neural network (CNN), providing a dependency tree where nodes represent the words, and the directed edges represent dependencies between the words. Each dependency is labelled with a relationship type that describes the nature of the dependency (e.g., subject, object, modifier).
We aim to test adding a syntactic understanding of the input sentence through constituency parsing, where the output reflects the syntactic structure of a sentence in terms of a hierarchical tree of phrases, sub-phrases, and words.
Additionally, we aim to test diferent types of neural networks widely used for NLU. Indeed,
as mentioned in section 2.4, Transformer neural networks have gained considerable attention
in NLP-related tasks, as they show superior performance in handling of context and sequence
than CNNs and recurrent neural networks (RNNs) [
Our KBQA system outputs the items in the DBT corresponding to a query. This raw dry output contrasts sharply with the user’s natural language query. Therefore, we decide to use NLG to represent the output in a more user-friendly way, testing template-based sentence generation and pre-trained language models (PLM) for text generation.
In both the NLU and NLG tasks, our goal is to find the optimal balance between the relevance of the output and the speed to obtain it.
To build and maintain our ontology and its alignments, we use Protégé12, an open-source tool from Stanford University that facilitates creating, maintaining, and editing ontologies. It ofers a graphical interface for modelling ontologies using OWL and supports testing inferences and running SPARQL queries.
Another output built from the work done by the domain experts is embedding properties and constraints into the DBT. Specifically, tools such as Autodesk Revit 13, ofer the capability to add custom fields, predefined values, and sometimes even some basic rules (e.g. EXAMPLE) to the model. This constitutes the preparatory step for practitioners to build and/or obtain a DBT enriched with our domain ontology.
To populate our KGs, the first step is to create an IFC-based repository in our graph DB (DataBase), loaded with the ifcOWL (IFC 2x3 TC1) ontology from the oficial file (using the turtle serialization format). The DBT is exported to an IFC raw file (STEP format), which is then transformed into an RDF file (using IFCtoRDF) to allow the model to be imported into the IFC-based repository of the graph DB. A small duplex apartment building contains 454,359 statements (364,216 explicit and 90,143 inferred) and has an expansion ratio of 1.25.
The second step is to create an NL-based repository in the graph DB, loaded with our OB27AI ontology aligned with IFC and the INLE ontology aligned with our ontology: both are imported from their respective RDF files. A small duplex apartment building contains 34,727 statements (18,856 explicit and 15,871 inferred) and has an expansion ratio of 1.84.
The third step is to build and fill an NL-based representation of the DBT from the IFC-based repository. Once our OB27AI ontology (aligned with IFC) is merged with the IFC-based graph representation of the model, the entities and concepts of interest are filtered based on the INLE ontology (aligned with ours), to fill the NL-based repository with the NL-enhanced model.
These three steps, shown in Fig. 6, are done using Python scripts and the GraphDB14 APIs (Application Programming Interfaces).
GraphDB, edited by OntoText, is our chosen graph database engine and knowledge management tool, equipped with reasoning and inferencing capabilities and compliant with RDF and SPARQL.
Python is selected as the programming language due to its versatility and the extensive array of libraries and frameworks available, both third-party and native, in the relevant areas.
The owlready215 module, licensed under GNU LGPL v3, is a Python module for loading, browsing, and modifying ontologies, as well as for searching, inferencing, and executing SPARQL queries.
The minimalist and lightweight Flask16 framework is employed for developing web services in Python. It provides the essential elements required to create lightweight web applications.
Conda17 is an open-source package manager that creates isolated Python environments, each containing packages and dependencies specific to various tools and experiments.
Project management and sharing are facilitated through GitHub, Microsoft’s online platform that uses the Git version control system.
Microsoft’s Visual Studio Code (VS Code) is the source code editor. It is lightweight, fast, and versatile, with particularly strong integration of Python, Conda, and Git.
Our approach for elaborating an NL answer from an NL query comprises 7 steps and is shown in Fig. 7.
The "Analyse" step uses the user’s question as input, triggering multiple NLU analyses to obtain one dependency and one constituency tree combined into a single tree representing the query.
The "Parse" step traverses the dependency and constituency tree to recognise potential entities. It organises them in branches of diferent lengths, following the overall structure of the input tree to generate an entity tree. This entity tree comprises all the possible branches between two entities, including their alternatives above a threshold based on semantic similarities with the terms from the origin sentence.
The "Ground" step traverses the entity tree to look for the entities in the NL KG while also searching for potential links between such entities in this KG. These grounded entities and links (aka grounded triples) are stored in a tree structurally identical to the one of the entity tree, though lighter as some branches have been pruned, either for the lack of possible relationships between entities or because their local relevance dropped below the threshold to keep them. The value for local relevance is computed on semantic similarities of entities and distances between classes and domains or ranges.
The "Extract" step explores the grounded triples, expands them in local SPARQL queries to be executed against the IFC KG, and ranks them against the triple score computed for the pruning in the previous step. The goal is to prune the grounded tree further when a query does not return a result, except for comparison (e.g., walls higher than 3m). Each branch’s best constraint (aka triples grounded in the IFC KG) is kept for the output list.
The "Generate" step stitches the constraints together in a shallow tree-like structure to account for the potential hierarchy of sub-queries to form the final SPARQL query.
The "Execute" step executes the final SPARQL query against the IFC KG and collects the raw list of IFC results.
The "Transform" step parses the raw list of IFC results to obtain the labels and types of the objects from the DBT so that very simple sentences can be created automatically using templates.
These sentences can be presented to the user, though more importantly, they can be used, in combination with the user’s query for context, to generate more complex sentences using PLMs (introduced in section 2.4).
As input to these models, either the keywords from the questions and the answers are provided, or the question along the template-generated answer and an instruction. For example, with the question "List the exterior doors with a width larger than 1m." (#21 in appendix A), there are 2 items in the ground truth: • The raw keywords are "2 door is external has property width larger than 1 meter". • The template-based generated answer is "There are 2 items of type door. These items are ’Door_6597’ and ’Door_6702’.". • For models that can incorporate prompts (i.e. Falcon, GPT, Flan-T5), the full input is then: "Question: List the exterior doors with a width larger than 1m.
Answer: There are 2 items of type door. These items are ’Door_6597’ and ’Door_6702’.
Generate a sentence to answer the question:"
For NLU tasks to understand the user’s query, the primary open-source library we used is spaCy18, renowned for its high execution speed and precision in processing text. Available for Python, spaCy is designed to handle various NLP tasks, including tokenization, lemmatization, parsing, and building dependency trees.
For the specialised and precise task of constructing constituency trees that represent questions,
the BeNePar19 (Berkeley Neural Parser) is used for its exceptional eficiency [
Hugging Face20 provides open-source libraries that ofer easy access to various pre-trained models for diverse NLP tasks alongside a platform for sharing models and datasets.
The tests were conducted on the two versions of our KBQA system, using a set of 22 questions like "Search for all the load bearing beams with slope equal to 0." (see appendix A for full list), the ground truth having been manually extracted from the DBT.
When the first version was evaluated against this new set of questions, longer and more complex, the recall dropped from 95.0% to 50.5%, and the F1 score from 0.974 to 0.671, mostly because this version has been using the small spaCy CNN model "en_core_web_sm", a model 35 times smaller than its transformer counterpart.
The two versions in our tests (see Table 1) have been configured to use the same NLP neural network from the spaCy transformer library ("en_core_web_trf"). The tests were performed on a computer with an Intel ® i5 CPU (2.5 GHz), 16 GB RAM, and the Windows 10 64-bit system.
Our precision is lower by 2.7 points due to many false positives in a single question (#11 in appendix A). The recall is improved by 18.6 points, while the F1 score, more balanced, shows an improvement of 8.9 points.
NSR-BIM v1 [
NSR-BIM v2 (Ours)
Variation (%) Recall Precision F1 score +10
The time taken for knowledge data preparation, shown in Table 2, is drastically reduced by 44%, mostly thanks to the use of a graph database instead of file-based storage.
NSR-BIM v2 (Ours)
Variation (%)
NSR-BIM v1 [
NSR-BIM v2 (Ours)
Variation (%) The query processing time, shown in Table 3, is further detailed below.
The step #1 is slightly longer as a constituency tree is produced in version 2 while not in version 1, although most of the time taken in the analysis of the question is linked to the NLP model used ("en_core_web_trf"): using a smaller and simpler model ("en_core_web_sm") results in a duration of 0.01s instead of 0.34s, though the trees obtained are syntactically and semantically too weak and faulty when longer and more complex sentences are used.
Step #2 in version 2 is 8 times longer (0.08s vs 0.01s) than in version 1 because two trees have to be parsed and combined, and mainly because semantic similarities against terms in the ontologies are computed at this stage.
The step #3 is new to version 2, as no grounding against KG was done in version 1. When this step was first added to the first version, it took 3.52s on average, more than twice the current time, as the existing tree traversing algorithm was not designed to handle that many tree branches and triples.
The step #4 is more than 4 times longer, as there are more extracted constraints, each tested against the IFC KG in version 2. Those constraints were taken as they were without further pruning in version 1: when introducing constraints testing against the KG in version 1, the average time climbed to 0.75s as too many constraints were tested.
Step #5 is twice as fast in version 2, as step #4 already performed the expansion of constraints into SPARQL sub-queries to enable executing them in the IFC KG.
The step #6 is almost 20 times faster, as version 2 uses a graph database instead of files.
Step #7 is new to version 2: the time measures have been done using template-based answer generation, not transformer-based PLM.
The overall time to process a query has doubled on average in version 2, largely due to the intermediate grounding of the whole tree against both NL and IFC KGs. For example, the question "List the exterior doors with a width larger than 1m." (#21 in appendix A), which used to have 2 branches with 12 triples in the first version, has 6 branches with 57 triples in the second version, each of them being tentatively grounded in the KGs in this second version.
Nevertheless, in version 1, this question #21 found no answers as the 2 branches completely separated "width" from "door": both branches were directly and separately attached to the root. This issue, widespread in test questions, was fixed by creating a larger and deeper tree representing the question: in the case of question #21, some branches were now linking both concepts of "width" and "door". However, having larger and deeper trees required better pruning to avoid the rise of wrong answers, hence the grounding and extraction steps (#3 and #4) that would test the branches and their triples against NL and IFC KGs.
Another example of a question that was failing in version 1 was "What are the dimensions of the door frame of all the building’s emergency exits?". This question is not part of the test set because version 1 could not possibly find its answer: indeed, this is the kind of question where a concept (e.g. "door frame") doesn’t exist explicitly in the model. Therefore, such a concept is inferred through our OB27AI ontology aligned with IFC, resulting in finding a relevant answer with our second version.
Regardless of implementation details, one major venue to improve the execution speed could be to use Cython, which allows compiling Python code in C or C++ and the direct call to functions written in C or C++. Another major venue could be loading the NLP neural network model to a GPU. There is also room for improvement in the grounding queries made for each possible triple.
A contextual performance comparison between our first version and MOP-SP [
To generate a user-friendly answer, we tested 5 models of Transformer-based PLM (see Table 4), available on-premises (as opposed to cloud-based), with both qualitative (rated "very poor", "poor", "good" or "very good") and quantitative (typical time to generate the answer) criteria. When possible, the prompt and the question in full were added as input to the template-based generated sentence.
For example, with the question "List the exterior doors with a width larger than 1m." the template-based generated answer is "There are 2 items of type door. These items are ’Door_6597’ and ’Door_6702’." and the PLM generated answers range from "Two door width more than 1 meter, exterior." (very poor) to "There are two exterior doors with a width larger than 1 meter, ’Door_6597’ and ’Door_6702’." (very good).
Integrating multiple knowledge sources and standards becomes increasingly imperative as the construction industry progresses into the digital era. This fusion blurs traditional distinctions between domains and emphasises the necessity of interoperability to facilitate seamless information sharing across disciplines. Moreover, the advent of Digital Building Twins (DBTs) represents a pivotal advancement in modern construction processes, enabling real-time data integration and supporting multiple architectural development phases. Our research contributes to this evolving landscape by presenting an approach that enhances the accessibility and interpretability of DBT data through natural language queries. By leveraging a domain-specific ontology and advanced AI techniques, our methodology facilitates eficient communication between humans and DBTs, allowing users to extract specific building details rapidly and accurately. Our comprehensive approach, encompassing knowledge representation, semantic analysis, and information extraction, represents a significant step forward in advancing the capabilities of DBT querying systems.
Future work Several strategies can be implemented when considering future work. Firstly,
grounding can be improved by setting a trust threshold to prevent systematic grounding.
Secondly, the number of NLP models loaded should be limited; for example, one could be
employed for constituency/dependency tree (NLU) and another for answer generation (NLG).
Additionally, maintaining context appears needed when accommodating multiple questions
about the same topic or object of interest. Further, extracting and sorting partial results can
provide explanations or clues leading to the final answer. To broaden the scope and robustness
of the system, an idea would be to expand test sets with questions sourced from practitioners
across diverse domains, along with generating various syntactic and semantic variations for
each. Lastly, as mentioned in section 4.2.2, aligning the OB27AI ontology with more existing
ontologies can contribute to its efectiveness and relevance across diferent domains while
enhancing the interpretability of the DBT, i.e. taking advantage of the topological knowledge
present in ontologies such as BOT (Building Ontology Topology) [
We want to express our gratitude to B27-AI for their financial and material contributions to this study and to the French National Agency of Research and Technology ANRT for their CIFRE subvention. Our thanks equally go to the Faculty of Science and Technology at the University of Burgundy for their academic assistance and to the anonymous reviewers for their valuable feedback. To evaluate the versions of our KBQA system, we used the following list of questions: