In recent years, advancements in digital technology revolutionized the way we document, preserve, and share cultural artefacts. Beyond any doubt, one of these tools is 3D reconstruction, comprising a vast set of methods for acquiring objects, from coins [1] to entire cities [2,3]. Such methods are largely employed in the cultural heritage domain for preservation [4,5], analysis [6,7,8], restoration [9] or dissemination such as virtual tours [10] or interactive visualisations [11]. Nowadays art and culture need to be accessible to everyone: an additional application of 3D reconstruction is to enhance accessibility of heritage objects for everyone. This can be done for example by making available digital content to users [12,13] or providing access to remote sites that are not easily reachable (e.g. underwater locations [14]). Another crucial part of inclusiveness focuses on individuals with disabilities [15,16]. This is often declined not only as producing the content itself or ensuring physical accessibility, but also in actively offering the same experience to people with imparities. Also in this case, technology offers a valid set of tools to implement this applications [17], enhancing the accessibility for a wide range of visitors categories. In this work we aim at embedding computer vision techniques directly in the 3D reconstruction and printing processes, with the final goal of adapting state-of-the-art saliency models to drive the printing process and enhance the experience of visually impaired people. This is carried out by exploiting the well-known set of techniques falling under the term of saliency detection and fixation prediction. people looking at the same subject. Additionally, we present a case-study involving the 3D scanning and printing of the Coats of Arms on display at Palazzo Bo (Padova, Italy). We scanned six objects and 3D printed them following the fixation map derived from the projection of the acquired surfaces. The main goal of the project is to create a reduced tactile "Coats of Arms wall", so that their significance and meaning are available for visually impaired people. A preliminary study including feedbacks from blind people shows the feasibility and the potential value of the project.
Accessibility for visually impaired individuals refers to the design of services, environments, and technologies that enable people with visual impairments to participate in society. Ensuring access to cultural heritage to visually impaired individuals can be implemented in several ways, for instance providing audio descriptions [18], accessible digital content with specific applications and technologies [19] or with tactile models. In [20] the authors propose a ring-like device to use while exploring a 3D surface so that the user gets in return an audio description according to the touched area. With a similar idea, the authors in [21] propose to track the user's gesture with a depth camera to guide the tactile exploration. In [18] authors propose to build 3D models and make them accessible to blind people via a haptic module, and in [22] authors developed a prototype in which blind users can explore an entire location combining tactile and audio descriptions. Another example is [23]. 3D printing is a widely studied technology that has been investigated for applications in cultural heritage domain for several purposes such as preservation, restoration or dissemination, just to name a few [24,25,26]. Some of these applications include accessibility for people with visual impairments. The work presented in [27] present a procedure for 3D printing specifically designed for blind people. In [28] the authors analyse scanning and printing techniques for the specific target of blind users accessing cultural content, while [29] presents an evaluation of the user experience with 3D printed replicas. In [30] the authors propose to increase accessibility of a permanent exhibition printing enlarged museum specimens to promote interactive and inclusive experiences. Other studies can be found in [31,32].
One of the challenges of accessibility is to develop a methodology to effectively create a presentation offering the same experience to different people. In particular, for visitors with visual impairments we have to exclude one of the most used senses for visual arts -sight. The question that follows is: what are the visual features that make us characterise an object? And are these features also interesting for a blind person? In this regard, we propose to address this problem exploiting visual saliency. When looking at an object, our gaze unintentionally lingers on some specific areas. Indeed, by tracking the eye movements while observing some subject, we can detect which regions are visually more interesting for our sight. Analysing the eye behaviour of many subjects observing the same scene allows us to compute the so-called fixation map. The concept of fixation map was introduced in 2002 by D.S. Wooding [33] and consists in defining a function that outputs the amount of visual attention for a given image location.
Following works aim at predicting fixation maps based on image features such as symmetry [34] or using data-driven approaches [35,36,37]. Since gaze estimation is closely related to human vision behaviour, fixation prediction models are often associated to salient object detection [38,39] or used to drive other tasks, such as classification or segmentation.
In this work we propose to apply state-of-the-art fixation prediction models to the acquired 3D objects and use the resulting fixation maps to drive the 3D printing. Starting from the acquired object with texture, we rotate the 3D mesh according a reference system and create a projection on a virtual plane that is perpendicular to the original object orientation. In this way, we can use the projected texture as input for the fixation prediction and identify the areas that would result more attractive for an observer. Exploiting the 3D acquisition of the objects, we can project back the visually relevant areas and adapt the printing process and some presentation aspects according to the fixation results. The main goal of the described process is to focus on the most salient object regions so that visitors touching the printed object can have a better understanding of the artefacts in all its parts.
Figure 1 summarises the proposed pipeline for acquisition and printing. First, the 3D scan of all the objects is performed, followed by some post-processing steps on raw data to improve the surface quality. The core part of our pipeline involves the application of fixation prediction on the acquired projected texture. This allows to effectively recognise the salient areas of the object that will guide the printing process. Finally, models are prepared and printed using two different technologies. In the remaining we describe a case-study where we exploited the attention mechanism for two aspects: first, we adapted the resolution of 3D printing according to the relevance; second, we focused on the most relevant regions and printed them separately with a different technology to offer a better reading.
Palazzo Bo is one of the most iconic buildings in Padova: its rooms are adorned with over three thousand heraldic Coats of Arms depicted in frescoes and carved in stone (see Figure 2, left). These objects represent people who held prestigious academic positions, therefore their presence offer unique insights into the history and culture of the place. However, traditional display methods limit accessibility for individuals with visual impairments. After an initial discussion with the museum staff, we concluded that reproducing the Coats of Arms was the most suitable choice for the project, for two main reasons: (i) Coats of Arms are omnipresent throughout the museum, adorning every wall and hall, so they are the most distinctive and prevalent feature; (ii) the museum staff usually face challenges in explaining the coats of arms to visually impaired visitors. We adopted two different 3D scanners for data acquisition, the EinScan Pro HD from Shining 3D (EinScan) and the Revopoint POP 3 from Revopoint 3D Technologies Inc. (POP3). The choice of using two similar tools derives from the intention of comparing a high-end instrument such as the EinScan (around 14, 000 Euros) with a low cost device (POP3 is around 700 Euros) with the idea that institutions with limited budget could possibly benefit from the same technique. Both devices are handheld and are able to capture the scene by manually moving the device around the object so that different points of view are acquired and automatically registered by the complementary software. The EinScan offers different acquisition modes: the HD mode offers an accuracy of 0.045 mm, and acquires 3000 points per second, while the Rapid Scan mode offers a maximum accuracy of 0.1 mm. The POP3 has a precision up to 0.5 mm at a working distance of 150 − 400 mm. Figure 3 gives an overview of the main post-processing steps, including noise reduction, point cloud alignment, hole filling and surface reconstruction, performed for each acquired object in order to obtain a printable mesh. The first step involves the removal of all points that are not part of the object itself, such as the background, then the following part consists in obtaining a watertight surface starting from the point cloud, i.e. generating vertices, normals and closing the holes. Finally, since the objects are fixed to the walls of the room, their back need to be reconstructed as a plane so that after printing it can be put on a horizontal surface. This is visible in the rightmost image of Figure 3, where we can notice the additional thickness added to create a planar base. After the characterisation of salient object areas, we adapted the 3D models, isolated the identified regions of interest, and proceeded with model preparation for printing. We decided to employ different technologies to print different areas of the objects and offer a better readibility according to the fixation maps (see Section 4.1 for details). In particular, we adopted fused deposition modelling (FDM) and stereolitography (SLA). The FDM technology was chosen to print the 3D model of the complete objects. FDM is a material extrusion technique in which a thermoplastic polymer filament is heated and a movable head proceeds to deposit the material layer by layer. We employed the Creality CR-10 Smart Pro 3D printer. This printer has a print size of 300 × 300 × 400 mm, offers a printing precision of ±0.1mm. In Figure 2 we show an image taken while printing a complete object with white material. The second technology we employed is SLA, used to print the surface details requiring an higher accuracy. It is a vat polymerisation method, wherein layers of a liquid contained in a vat are successively exposed to ultraviolet (UV) light. The liquid material reacts to incoming light, resulting in curing only the areas exposed to UV and causing selective solidification. We used the Formlabs Form 3 printer, characterised by a laser spot size of 85 microns, by a build volume of 145 × 145 × 185mm and a layer thickness of 25 − 300 microns. Figure 2 shows the completed print of a selected inscription detail: the object grows layer by layer from top to bottom, and thus a support structure in this case is needed while the printing proceeds. Usually, a higher number of points suggests a higher accuracy: looking at Einscan acquisitions, we can observe that objects A, B and D have ≈ 1𝑀 points, while object F has ≈ 6𝑀 points due to the HD mode that was selected only for the last object. Regarding the POP3 acquisitions, objects C, D and E exhibit roughly half the point with respect to the other objects, denoting a lower surface resolution. Object D was acquired with the two scanners to assess the feasibility and analyse possible limitations of different devices. The EinScan shows a higher resolution, while the surface acquired by the POP3 is smoother and exhibits a less marked inscription. Despite the inherent challenges of manual acquisition, the POP3 managed to yield satisfactory results, largely attributed to the capabilities of its software (Revoscan 5), which played an important role in refining the acquired data. After acquisition, we used the acquired models to generate 2-dimensional texture projections on a plane and obtain an RGB image.
Figure 5 shows the images used as input for visual saliency. We applied the fixation prediction method as proposed in [37] and used the original weights as provided by the authors. The resulting visual attention is shown in Figure 5, applied to two of our objects. We plot fixation maps with a colour scale representing different levels of attention, where 0 value means no attention and 1 indicates the maximum attention. The third and sixth images of Figure 5 show the masking applied to the objects according to the attention map, so that we have a clear interpretation regarding the most salient areas on the objects surfaces. We can notice that for all the analysed objects, we can identify two to three interesting areas exhibiting the highest attention, depending on the individual object features. For all items, one part resulting particularly interesting for our sight is the central part of the Coats of Arms, depicting the symbol representing its owner. Another interesting area for objects A and B is the small cherub on the top, while for other objects (D, E, F) the upper areas do not result particularly relevant. Finally, for some objects (e.g. item F) also the bottom part with inscriptions results attractive.
We concluded that the central parts of the objects need to be printed with higher details and also to be highlighted during presentation. Also, we focused on the printing of the cherubs and the inscriptions on the lower parts of the objects to improve readability. First, we printed the entire objects using FDM printing technique setting the layer height to 0.2mm and an infill density of 15%, Figure 6 (left) shows an object printed with PLA: the overall quality is good, except from some flat areas in which the printing layers are clearly recognizable. In particular, this is quite evident in some details (see Figure 6, center), where the resolution results altered by printing layers. Following these observations SLA printing was adopted to reproduce regions with the highest visual attention. We used Grey v4 resin, well-suited for general-purpose prototyping, particularly for models demanding intricate details, similar to ours. Figure 6 (right) shows a detail printed with SLA technology, offering higher resolution and a better understanding of the underlying surface details.
As a preliminary result, we involved in a survey two individuals with visual impairments who volunteered to provide initial feedback. The main goal of the session was to determine the objects general usefulness, and also to assess the quality improvement offered by the direct application of visual attention prediction. During the survey some challenges were identified, particularly regarding the initial comprehension of the objects and the influence of printing layers on readability. In particular, the perception of printing layer for FDM is significative, and brings the need of explaining that they are not part of the object. Also, differences between FDM and SLA printing were noted, suggesting the need for refinement of printing techniques to optimize tactile perception. Regarding the effectiveness of the visual attention approach, during the survey we took note on which surface regions resulted more interesting from the tactile point of view, and we observed that the regions highlighted by the fixation prediction were the most attractive surface areas during the tactile examination. Moreover, the participants appreciated the SLA-printed details as helpful means to improve their understanding of the whole object. Overall, the project was deemed useful in providing tactile representations of the Coats of Arms, facilitating comprehension and engagement. As a future work, we aim to extend the survey in a more structured way, collecting feedbacks from a wider range of people, and performing an extensive study about object readability driven by visual attention and fixation prediciton mechanisms.
In this paper we propose to merge 3D reconstruction and printing techniques with computer vision algorithms to enhance the experience of visually impaired visitors. We present an attention-driven method which exploits 3D scanning of artefacts and applies to the printing process of cultural heritage content. A preliminary study involving a survey highlights the effectiveness of the method, giving a strong direction for future improvements and investigations of the proposed method.
We acquired 6 coats of arms: Figure 4 shows all of them with their identifiers. Table 1 summarises the final results in terms of acquired points (raw data) and number of triangles for each object and device.
This study was funded by the European Union -NextGenerationEU, in the framework of the iNEST -Interconnected Nord-Est Innovation Ecosystem (iNEST ECS_00000043 -CUP H43C22000540006). The views and opinions expressed are solely those of the authors and do not necessarily reflect those of the European Union, nor can the European Union be held responsible for them.