Introduction

Virtual reconstructions have the amazing power to bring a past to life. A virtual reconstruction is a common exercise for the disciplines interested in cultural history to represent some aspect of a past reality. However, the resulting reconstruction images and 3D visualizations conceal the underlying scientific process and hypothetical reasoning that contributed to their production. In this article, we tackle the gap in the literature between the knowledge that is produced in a reconstruction study and actual reconstruction data that one can either query or visualize. In the context of Notre-Dame de Paris, the fallen transverse arch in the nave provides a case study (Figure 1), where the challenge is to determine the positioning of its voussoirs before its destruction. The 3D visualization prototyping and querying bring to the foreground the hypotheses and argumentation, where the challenge is to determine the positioning of its voussoirs before its destruction. Given the fragmentary documentation available, there are several plausible solutions to reconstruct this arch, some more valid than others. We then need to present the reasoning behind these choices. Such a case study allows for testing a data workflow on the reconstruction argumentation reasoning.

The question of virtual reconstruction can be defined as a scientific study that aims at formulating an argumentation linking three elements: (1) the material objects (voussoirs, arch), (2) the digital objects (pointclouds, digital photographs, datasets), and (3) expert knowledge (archaeology, art history, architecture, computer science etc). These elements are tied together with the hypothetical scientific reasoning about reconstruction question. In this work, we posit that the 3D visualization and querying are constitutive research tools for the virtual reconstruction research, rather than an end result medium for dissemination. By letting the voussoirs speak, we mean that the materiality of the voussoirs and immateriality of digital surrogates are the support to make explicit the argumentation of the reconstruction. The virtual reconstruction research is rooted both in the real (material and immaterial object) and the unreal (hypotheses formulation), where the scientific inquiry about the past can be developed. Then the challenge is about reasoning both about factual knowledge and counterfactual propositions: a reconstruction formulates a hypothesis about something that might have existed based on the existing presence of objects that are surviving witnesses of this lost past.

This contribution proposes a twofold experiment. Firstly, it sets up the 3D dataset of Notre-Dame's de Paris' transverse arch reconstruction, including the data enrichment with archaeological knowledge proposed by archaeologists and subject experts. These requirements were detailed in previous studies [1,2,3,4,5,6]. Far from being straightforward, this step introduces the innovative elements of linking counterfactual reasoning to real things (physical or digital) and unreal things (competitive reconstruction hypotheses) at the dataset level. The scientific narrative of the reconstruction study is tested using Prolog programming, where the formal expressions of relations between hypotheses, arguments, and objects are modeled. Secondly, after the transformation to Prolog facts and rules of the initial reconstruction dataset, the evaluation of the scientific narrative of reconstruction argumentation is tested through both the 3D visualization and query of the 3D enriched data. The humanistic question of reconstruction is the starting point for In section 2, we present the existing gap in documenting counterfactual reasoning of virtual reconstruction. The scientific inquiry over reconstruction leads to formalizing concurrent points of view about space, place, and memory, that bring the theoretical concept of place making. Querying and visualizing reconstruction data opens the problem of representing factual/counterfactual and spatio-temporal data. In section 3, we develop the experiment's methodology: the reconstruction hypotheses are considered as versions or knowledge steps. Subsequently, in section 4, we detail the implementation the counterfactual layer in the reconstruction data using Prolog. In sections 5 and 6, the complementarity of the 3D visualization and querying for the understanding of the reconstruction reasoning is demonstrated: in section 5, the resulting transformed data is evaluated with the competency questions focusing on the counterfactual elements, while in section 6, the 3D visualization shows how the hypothetical reasoning of the arch reconstruction is embedded in the spacetime volumes of the voussoirs.

State of the Art

Reconstruction in Archaeology: the Problem of Documenting Counterfactual Reasoning

In this part, we present the gap in the literature between the knowledge about the past in a reconstruction study and the actual reconstruction data. The problem of documenting virtual reconstruction has roots in philosophy of language, semantics, and logic. For Clark [7], the reconstruction of the past in archaeology is a misnomer since virtual reconstruction is rather the construction of a model for the study of the past as "tools for understanding, not statements of reality". This debate is exacerbated with the development of virtual archaeology and the democratization of digital documentation [8,9,10,11]. A reconstruction study is "based on complex chains of reasoning grounded in primary and secondary evidence that enable a historically probable whole to be reconstructed from the partial remains left in the archaeological record" [9]. In a previous work [4] we demonstrated that these complex chains of reasoning in virtual reconstructions experience a phenomenon of compression both at a level of representation and reconstruction. The reconstruction study builds on knowledge of real/"actual" things (artifacts, remains etc.), as well as unreal or "intricate counterfactual blends" [12].

For memorizing the reconstruction reasoning, the problem can be formulated as follows: the reconstruction study (or model) formulates implicitly a knowledge object that is in-between real objects and counterfactual propositions. Making the voussoirs speak, we aim at unfolding and explicitly formulating these intricate counterfactual blends. The goal is the explanation of the proposition sets, arguments, and inference logic that are used in the argumentation of the reconstruction [4]. From the perspective of data science, Pearl [13] posits interpretation as how to connect the reality and the data, as a way to encode causal assumptions. Transparency and testability are proposed as requirements for data transformation: "Transparency enables analysts to discern whether the assumptions encoded are plausible (on scientific grounds) or whether additional assumptions are warranted. Testability permits us (whether analyst or machine) to determine whether the assumptions encoded are compatible with the available data [...]" ( [13], p.58). Our proposed experiment enriches the reconstruction data and encodes the reconstruction arguments as the causal assumptions used in the virtual reconstruction, applying the transparency and testability criteria.

Formalizing the Concurrent Points of View about Space, Place, Memory, and Placemaking

We explore the problem of the reconstruction of historical buildings and their restoration, considering a series of competing interpretations. This leads us to look at space as meaningladen, that is a place. Nora introduced the notion of lieu de mémoire (place of memory) [14], which can help in formalizing our understanding of the interrelations between space and memory. Lieu de mémoire, as defined by Nora, is not just a physical location, it embraces the "embodiment of memory in certain sites where a sense of historical continuity persists" [14]. The lieux de mémoire exist because of the disappearance of milieux de mémoire (cultural material environment). The virtual reconstruction of the transverse arch is consecutive to its destruction. The voussoirs have survived and became the material witnesses of the destroyed arch. Lieux de mémoire are the surviving, remaining material elements of embedded memory. Memory, either social, collective or individual, is affected by its relation to space and materiality: its meaning attachment to a space creates a place. Stories and events play an important role in making a space a place. For every place, different people have different associated memories [15,16], often providing a subjective version of the same place and the events that occurred. Number of research works in records and documents oral, sensorial, and written histories associated with places. Place encompasses materiality that ranges from the scale of objects, buildings to urban fabric or landscapes. That explains why urban planners, historians, architects, and anthropologists give a significant importance to the concept of placemaking since memories give meaning to the surrounding materiality. At the urban scale, memories may be available in several forms [17], media documents forming a key form of recording. The documents narrating the historical events and their chronology frequently serve as evidence for understanding the incremental changes that happened to urban and architectural objects. Historical documents and archives have been used with their 2D/3D representations [18,19] to represent concurrent points of view of urban evolution capable of narrating numerous possible scenarii of city changes at different levels of detail. Numerous ontologies and documentation standards have been proposed to represent (historical) episodes, events (CIDOC CRM as event-centric ontology), stories, historical narratives [16], and argumentation using CIDOC CRMinf [20] or other argumentation models [21].

With the models and ontologies in place, a methodological approach based on versioning is used to store and visualize historical narratives about cultural heritage artifacts at different scales. Extending the use of version control systems like Git, Samuel et al. [22] propose to store multiple scenarios of urban evolution as city versions. Hypermedia stories -graph-structured stories with navigation links - [23,16] uses media along with these narratives to better explain the modification of the urban fabric and cultural heritage assets. At the architectural scale, the evolution of objects is apprehended through diagrammatic visualizations called diachrogram to capture the changes and periods of stability in building lives [24]. Metral et al. [25], Bruseker et al. [9] show the compatibility and applicability of knowledge representation and ontology design approach respectively to the urban and built heritage domain.

In this article, we want to go beyond the use case of representing and visualizing urban evolution, as well as the linear [26,27] or the graph-structured stories [28,16] of a series of events [29]. We propose the extension of some of the above works to represent the hypotheses evolution during the reconstruction process. We build upon the studies on historical changes to model the hypotheses associated with virtual reconstruction, where the goal is to ultimately bring back a damaged object to its hypothetical lost state.

Visualization

For the visualization in heritage field, one observes a duality between semantic-oriented approaches and 3D visualization-oriented approaches. On one hand, the approaches centered on knowledge modeling mainly present information systems designed to manage documentation activities, without directly exploiting the full wealth of heterogeneous data mobilized in heritage contexts, in particular 3D data [30,31,32]. On the other hand, approaches based on the exploitation of 3D digital data, focus mostly on their description, omitting any reference to the complex relationships specific to the observation, interpretation, or documentation processes [33,34]. Some works, such as Soler et al. [35], Apollonio et al. [36], clearly demonstrate the strong interest in bringing these two approaches together, making it possible to spatialize and query heterogeneous information in the context of restoration operations. However, these initiatives are generally built around isolated 3D models, excluding the possibility of handling and confronting the diversity of resources involved in heritage studies, like Notre-Dame scientific action.

From a software point of view, there are a number of very effective libraries dedicated to the manipulation of 3D data, which can be used for reconstruction studies [37,38,39]. But these tools focus on visualization and interaction modalities, allowing only in the best cases to display provenance metadata, and in most cases ignoring the full potential of an upstream work on data curation and conceptual modeling. In [40], spatio-temporal information of 3D data is characterized describing the transformations affecting buildings over time and considering their changes (demolition, amplification, division, or change of function). It is based on the addition of a semantic layer to the geometric restitution through description graphs, enabling information on buildings to be enriched according to existing documentary sources. While this approach is of considerable interest, it remains ill-suited to virtual reconstruction data documenting the scientific context and reasoning about the reconstruction process. The temporal dimension addressed in [40] is linear, making it difficult to semantically and geometrically characterize competing hypotheses and their underlying arguments. Samuel et al. [19] and Jaillot et al. [41] propose knowledge versioning in 3D urban contexts. While this approach is interesting in terms of data structuring, all the narrative potential remains to be explored.

Methodology

In this section, we consider the example of the arch of Notre-Dame de Paris that was damaged during the 2019-fire. The voussoirs that together formed the arch fell on the nave's floor and were displaced. The scientific research and restoration activities have divergent objectives and interpretations, while working on the same object. The virtual reconstruction study aims at gathering knowledge about the destroyed arch and understanding which original place or slot was for each authentic voussoir in the arch before the fire [3,5,6]. However, this process is extremely difficult and several sessions were held [1,2], where historians discussed and proposed reconstruction hypotheses. Figure 2 shows multiple rounds of discussion for placing a voussoir in specific slots. Six overall hypotheses were formulated at different times for identifying the position of voussoirs. It is important to note that a given voussoir may have changed their slots during the different hypotheses based on additional collected evidence. The ultimate goal is to possibly relocate the voussoirs and the arch to their hypothetical original positions on time T1 (i.e., before the fire). Unlike [18] or [19] that look at the evolution of historical artifacts or city objects (T1, T2, T3) as recorded in time, the reconstruction uses a fiction as if we were looking backward in time. In fact, the past object of study (T1) is a hypothetical model for knowledge, where the reconstruction hypotheses can be seen as versions. At the time T1, there are several hypothetical versions that were proposed with the end goal of creating a reconstructed arch that gets as close as the arch before the fire (i.e. the hypothetical original version V0 at T1). It must be pointed out that we may never arrive at the ultimate version of the reconstruction V?? (shown in Figure 2 ) that closely approximates V0. In order to restore the arch to its hypothetical original state, it is critical to understand that the arch has two sides, the northern and southern sides. A number of 122 voussoirs together form the whole arch with 40 voussoirs still in place and 77 recovered voussoirs. The (possible) slots for the fallen voussoirs have been identified and numbered, using survey documentation prior to the fire. The slots on the south side are numbered with even numbers, while those on the north are numbered with odd numbers, starting with R0 for the keystone. Thus, it is easy to understand that R12 is the 6th slot on the north side of the arch. The fallen voussoirs have also been identified and inventoried N1,. . . , etc.

The goal of reconstruction is to assign a slot 𝑅 𝑛 (values of n are between 0 and 121) to every voussoir 𝑁 𝑘 (values of k are between 0 and 121). There are multiple collaborative work sessions between researchers (archaeologists, architects, art historians, etc.) to produce such possible reconstruction hypotheses. Each hypothesis has its associated arguments.

In Figure 2, the arrows between the two versions may have different significance for the reconstruction reasoning. A decision about a voussoir based on a particular argument may remain valid across different versions. However, this also means that any initial hypotheses, if proven false at a later date may lead to the falsification of some pairs of slotvoussoirs from another hypothesis. Take, for example, the evolution of the argumentation process related to cross-shaped marks on the joint sides of the voussoirs. These lapidary marks constitute an archaeological predicate for the arch reconstruction. The marks were believed to possibly serve as guide the masons to orient the positioning of the voussoir during the construction. The mark could indicate either the downward or upward side. In a first hypothesis the voussoirs were oriented with their lapidary mark facing downward. This was later proved to be incorrect. This meant that we had to relook all the previous use of these arguments and make corrections to the assigned facts: slot-voussoir pairs, voussoirs, if necessary. This makes the problem particularly challenging since we need to record this aspect for every voussoir and the assigned slot along with the associated arguments in the different hypotheses. In the reconstruction data, we have a selection of relevant information about the physical objects (voussoirs) and possible locations (slots) in the state of the arch before destruction. The difficulty lies in the reasoning about the characteristics about the real objects in relation to the argumentation hypothesis of the reconstruction, i.e., counterfactual reasoning. The reconstruction data is an open-ended hypothesis, as shown in Figure 2.

Implementation

After exploring the initial reconstruction dataset available in [3], the experiment proposes to encode the reconstruction arguments as the causal assumptions used in the virtual reconstruction. It was important to represent them in a format so that we could easily perform analyses, querying and reasoning, applying the transparency and testability criteria [13]. The first step consists of the representation of reconstruction hypotheses and the associated metadata as RDF. The full list of information of voussoirs and slots is available in textual format [1,2,3]. We represent essential information about the proposed locations of the voussoirs along with the arguments for this hypothesis (Tables 1 and 2).

Prolog is a versatile declarative programming language that allows a set of facts and rules (e.g., the details of a voussoir or the details of a hypothesis) and rules (e.g., linking hypotheses to voussoirs and proposed slots), which define relations [42]. Prolog has been chosen, considering its declarative nature and as a useful language for quick prototyping and short iterative tests on the data. It allows inference and reasoning, data validation and verification. The dataset has been transformed as facts and rules

No. Predicates in Prolog Description

1.

𝑣𝑜𝑢𝑠𝑠𝑜𝑖𝑟(𝑁, 𝑊, 𝑋, 𝑌, 𝑍, 𝐹 ) information related to a voussoir: N, its identifier, W, the width, F, the face where the crossmark is present (up or down), the center of gravity given by X,Y,Z.

2.

𝑠𝑙𝑜𝑡(𝑅, 𝑋, 𝑌, 𝑍) location of the slot with R its identifier and X,Y,Z the center of gravity (useful for visualization).

3.

ℎ𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑠(𝐻, 𝑅, 𝑁, 𝐶) details of a hypothesisl H linking the voussoir N to the slot R, with the comment C.

Table 1

Some key reconstruction predicates that were used for querying and inference.

in Prolog (Table 1) to explicitly express the argumentative relations between voussoirs, slots, and hypotheses. We could then apply the testability criteria on the counterfactual reasoning of the reconstruction to verify whether there are incoherent elements in the data. Additional information is inferred about the reconstruction reasoning from the existing data: for any voussoir, we know the sequence of hypotheses that a voussoir went through. Instead of tracking events (temporal data) of the reconstruction work sessions, we are more interested in modeling the hypotheses about past states (Hypothesis [N:1-6] in Figure 2). To say it differently each voussoir can tell its own story in the reconstruction.

As in [19], another facet of the reconstruction reasoning is inferred: the transition (as event) between two versions (as stabilized state of an hypothesis) are made explicit, e.g., when a voussoir was deleted, or replaced in a given slot or when a new voussoir was replaced in a slot. A sequence of hypotheses and hypothesis transitions can be considered as an overall story, i.e., one possible narration of the reconstruction. We move from the known documented state to the implied events that lead to it.

Evaluation using competency questions and Prolog queries

The objective here is to go beyond commonly found linear temporal visualization, where a temporal bar shows the state of an object at different time spans. Instead, archaeologists and subject experts want to go at a more granular hypothesis versioning level. Their focus ranges from individual objects (voussoirs) or the overall main structure (arch) or a combination of both. It opens up parallel narratives about the arch, representing the virtual reconstruction hypotheses with the argumentation about the authentic voussoirs as archaeological artifacts. The virtual reconstruction process is not just about the story before and after the reconstruction, but the narrative in-between. The study of different possible arguments used in this process serve as lessons for future research.

To evaluate the experiment, a list of queries are formulated as competency questions (Table 2). For example, the analysis (based on semantic, temporal, spatial elements) helps in understanding the case of a voussoir staying at the same place across certain hypotheses or all hypotheses. A subset of the research questions have been formalized as competency questions and translated using Prolog queries. The Table 2 shows how to

No. Competency questions Query (Prolog) 1.

What do we know about all the voussoirs and their associated information (N as the identifier, W as the width, X as . . . )?

𝑣𝑜𝑢𝑠𝑠𝑜𝑖𝑟(𝑁, 𝑊, 𝑋, 𝑌, 𝑍, 𝐹 ).

2.

What do we know about all the reconstruction locations for the voussoirs, ie. the slots and their related information (R as the slot identifier, W as the slot width, . . . )?

𝑠𝑙𝑜𝑡(𝑅, 𝑊, 𝑋, 𝑌, 𝑍).

3.

What are all the hypotheses? ℎ𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑠(𝐻, 𝑁, 𝑅, 𝐶).

4.

What are the pairs of voussoirs and the associated hypotheses?

Table 2

Some competency questions to query the reconstruction data using Prolog fact and rules in regards to the reconstruction question.

use the predicates formatted in Prolog to answer the reconstruction questions. Finally, the validated data is then visualized in the 3D viewer for visualization and evaluation of results, as shown in the next section.

Results: 3D visualization of the reconstruction narrative and hypotheses

The reconstruction hypothesis is projected in the real existing space of Notre-Dame de Paris cathedral. This is the reason why it is equally essential to verify the reconstruction reasoning using 3D visualization and query-based data validation approaches. Once the required responses were obtained using Prolog, these were then visualized in the custom 3D nd-Viewer [43], allowing the use of multi-faceted variables in their spatial context (like semantic information including links to the current or past versions). The objective is to obtain a virtual equivalent of these results and to visually verify them. The 3D visualization is a complementary approach to querying. Whereas Prolog gives information concerning the way that data and causal assumptions are encoded and described, the 3D visualization investigates the relation of digital place making as the relation between space, time, semantics, and memory. It is one thing to note that the position of a keystone is never questioned and remains stable between several hypotheses, and another to see it in context and understand that this stability can be explained by the spatial or geometric relations linking this keystone to its direct environment. The concurrent states of the same voussoir in different hypotheses can be highlighted and compared, since the reconstruction reasoning is revealed in the 3D viewer. Thanks to inferred data about reconstruction reasoning (state of hypotheses + events of change), the 3D viewer can animate the voussoirs in regards to the reconstruction reasoning as trajectory. Unlike 3D visualization with temporal bars, where the different states can be simply visualized, the goal here was also to obtain a smooth animation of transition of these hypotheses for one voussoir. Several problems arise: the orientation of the voussoir when it changes the slots and orientation of the trajectory. For example, in Figure 3, we fix the visualization of the voussoir in one slot and show its possible trajectory across different hypotheses. We can see that each point corresponds to the different hypotheses. The visualization also shows the archaeological information for the voussoir for further validation by subject experts. It is also possible to glide the same voussoir in different slots across hypotheses.

Discussion

Virtual reconstruction of Notre-Dame's arch is a complex question, several rounds of hypothesis sessions have been conducted since 2020. The data does not represent a finished process: the ongoing input of new data leads to new arguments that will themselves enrich further the argumentation of future hypotheses, as shown in figure 2. Because it is an interdisciplinary research, it ensues that communication is a crucial aspect. We require different ways to communicate, document, and visualize the evolution of these hypotheses about the reconstruction. Neither 3D visualization nor formalizing/querying may not be enough separately. A data spreadsheet, obtained from inference tools, like Prolog, lacks readability without visualization. 3D visualization, on the contrary, without data formalization and inference, is missing the depth of reconstruction knowledge.

The combination -formalization, reasoning, 3D visualization -helps to demonstrate the hypotheses and can also be used to explore new questions that were inaccessible previously. For example, we can assess whether a high degree of variability between different hypotheses can still exhibit some form of local stability (e.g., voussoirs whose position have been much questioned but which are still stuck together as clusters). The questions on semiotic visualization still remain open: the semiology about 2D graphics is part of the general culture due to geography and architecture, while the 3D graphics mostly represent a mimetic relation with the reality. The codes and symbols for representation in 3D are not yet conveying interpretation and narration, which impacts the cognitive engagement and interactions from the users. They are yet to be invented and explored as spatial patterns.

Conclusion and Future Works

This article presented the study, representation, and visualization of the hypotheses elaborated for a virtual reconstruction study, especially after a destruction event like in Notre-Dame de Paris. A number of past research works on cultural heritage, storytelling, and urban evolution have previously explored models, methodologies, or ontologies for narrating linear or concurrent scenarii of evolution. In this article, we presented a different case, where the goal is not to obtain the past evolution of objects, but to build knowledge about objects in a hypothetical state. The granularity of argumentation is still to be explored, i.e., the possibility to find the most granular arguments (or atomic arguments) upon which further arguments can be built.

We proposed a quick prototyping and short iterative tests on the data for exploration. Making use of the hypotheses' data, both logic and visualization tools were tested. It allowed us to track the different hypotheses, and rediscover the reconstruction problem under a new light. The combination of formalization and 3D visualization changes the paradigm for reconstruction scholarship. The need goes beyond data provenance and reusability. It opens avenues for exploring data intensive collaborative environments. The experiment nourished our initial intuition that serendipity is a key component for data visualization. It allows the users to navigate and interact with the 3D objects and discover some additional characteristics that are inferred via reasoning over the data. The experiment shows that 3D visualization setup opens the possibility for spatial serendipity as spatialized query of embedded memory in place.