Cultural heritage plays a vital role in preserving the memory and knowledge of the past. Moreover, its preservation is essential in developing modern infrastructure, constructing new cities, roads, and railways. At the same time, we must not forget about the development of tourist services, an adaptation of old buildings to modern needs, illegal archeological excavations, and other potential risks related to destroying cultural heritage.
Preserving cultural heritage has three main risks. First of all, it is a time-consuming process that is required to be done repeatedly. If it is impossible to do so, cultural heritage will get damaged and require renovation, or in some cases, it can even be lost forever. The second concern is financial resources. Non-stop monitoring takes much time and human resources; thus, it takes much money. No monitoring results in cultural heritage damage, and restoration takes even more money. The last but not least risk are people working with cultural heritage. They do a monotonous job checking cultural heritage for destruction. Instead, they could spend time on research and renovation tasks.
Nowadays, cultural heritage monitoring is managed by cultural organizations, which are constantly confronted with a large amount of data that needs to be processed and small resources that they can use. The solution is to create a risk-informed system to automate data monitoring and analysis based on 3D and AI technologies. By automating the processes in collecting and analyzing information, it is possible to achieve significant cost savings, both human and financial.
The work aims to systematize approaches to directions and technologies of 3D and AI technologies to analyze and recognize cultural heritage, developing a system for practical application.
The solution of the following tasks is required:
1. Review of existing 3D and AI solutions for monitoring and analysis of cultural heritage preservation. 2. Research of requirements, methods, and algorithms to get a decision for the task. 3. Select and collect the necessary data of cultural heritage to be analyzed. 4. Development of architecture for monitoring and analysis of cultural heritage preservation. 5. Creating an application program -a system of semantic segmentation for cultural heritage.
The study's practical importance is to create a new risk-informed system for functional, objective, and cost-effective monitoring of cultural heritage changes that will monitor many facilities and act quickly and promptly.
Risk analysis is one of the essential tools for preserving cultural heritage. It is used for decisionmaking in the process of cultural heritage asset management and maintenance. For this purpose, both quantitive and qualitative analysis is used [21].
Although risk categorization plays a vital role in risk management in other disciplines, it has yet to be successfully applied to cultural heritage studies [22].
The importance of information searching and systematizing in the modern world has led to the thematic modeling of text document collections in this study. Therefore, thematic models are used to identify trends in scientific publications or news streams for classifying and categorizing image documents and video streams, information retrieval, including multilingual, tagging web pages, detecting spam, recommendation, and other applications.
3D scanning -building a computer model of a material object. It is studied by many researchers [5][6][7][8]. Currently, there are two leading technologies of 3D scanning -laser scanning and photogrammetry.
Laser scanning is a technology for obtaining information about terrain and objects using a laser. This method has been studied by scientists [5][6][7][8]. The result of the laser scan is a cloud of laser reflection points.
There are two types of laser scanning: mobile and stationary. During mobile scanning, continuous measurement is performed while the vehicle is driving. The device is established motionlessly during stationary scanning, and measurement is carried out from several standing points.
Photogrammetry is a science that studies the appearances, shapes, and positions of various objects in space, objects, and their shapes by measuring their photographic image. It was studied by researchers [5][6][7][8].
No special devices are required to use this method. It is enough to have a camera on a modern phone.
When choosing photogrammetry for 3D scanning, an important question is what and how affects the accuracy of 3D models. Accuracy can depend on many factors [1]: optical and digital camera characteristics, spatial distribution, and ground control points.
Also, shooting with the help of remote control systems, such as drones, has been studied. This method has an advantage over standard photogrammetry due to the aerial view [2].
In recent years, interest in preserving cultural heritage has begun to grow, so more and more data is being digitized, which is very important for artificial intelligence, as the training of models is based on data. However, collecting a large enough amount of data is still a problem because it is time-consuming and requires a human factor to mark the correct elements.
Machine learning technologies have become popular not only in computer science but in other fields as well. One of the reasons for the growth, in particular, is the successful application of deep learning methods for image classification [3], in which convolutional neural networks (CNN) exceed the human ability to analyze objects [4].
The potential of deep learning technologies for three-dimensional image analysis achieved a remarkable breakthrough in 2012 when the AlexNet model [5] showed excellent analysis results during the ImageNet competition. In 2014, GoogLeNet [6] won the ImageNet competition, achieving 93.3% accuracy of semantic segmentation. In 2016, Microsoft Networks ResNet [7] won the ImageNet competition, achieving 96.4% accuracy.
Learning on point clouds is attracting more and more attention with the development of augmented and virtual reality, their wide application in computer vision, autopilot, robot development. Deep learning is well researched to solve 2D problems, but for 3D cultural heritage data, it is only evolving and needs further research and the creation of new datasets to train effective models.
In this paper, 3D data are presented in point clouds, and the task is their semantic segmentation. For this purpose, the rights to use the ArCH dataset (Architectural Cultural Heritage point clouds for classification and semantic segmentation) were obtained. [8] The dataset consists of 17 annotated scenes, each point of which belongs to one of 9 classes: "arch": 0, "column": 1, "moldings": 2, "floor": 3, "door_window": 4, "wall": 5, "stairs": 6, "vault": 7, "roof": 8, "other": 9. Some of these scenes belong to the UNESCO heritage. Others are part of the historical heritage and represent different historical periods and architectural styles.
Fifteen scenes are used for training and two for testing models. Scenes for training include churches, chapels, porticos, loggias, pavilions, and monasteries. Two test scenes have different characteristics. The first represents a simple, almost symmetrical one-level building with standard and repetitive geometric elements. The second scene represents a complex, asymmetrical building with two levels, shot both inside and outside, with different types of vaults, stairs, and windows. Point-based Networks are used for the segmentation (Fig. 4).
The semantic segmentation task is to divide a cloud of points into parts according to the semantic meaning of the points. This section will describe the best semantic segmentation techniques nowadays: Point-wise MLP on PointNet [9], PointNet ++ [10] and RandLA-Net [11], Point Convolution on PointCNN [12], RNN-based on RSNet [13], Graph-based on DGCNN [14].
Pointwise MLP methods typically use common MLP (Multi-Layer Perceptron) as the central unit in their network for its high efficiency. However, point functions obtained with MLP cannot cover local geometry in point clouds and interactions between points. Therefore, various methods have been proposed, including PointNet, PointNet ++, and RandLA-Net, to provide more context for each point and explore deeper local structures.
Convolutional networks require highly structured data to obtain scales and other optimizations. Because the point cloud is not standard, the data must be transformed into a voxel grid or image collection before transmitting it for learning. However, this transformation makes the obtained data excessively voluminous and can also change the nature of the data. That is why PointNet accepts a cloud of points without transformations.
The PointNet architecture (Fig. 5) consists of three main modules: a max-pooling layer as a symmetric function for aggregating information from all points, a structure for combining local and global information, two networks for combining entry points and point features. The idea of this model is to approximate the general function defined on the set of points through the application of a symmetric function on the transformed elements in the network (Formula 1):
where
-is a symmetric function. H is approximated through the MLP network and g through the composition by a function of one variable and max pooling function.
PointNet does not cover local structures because of the metric space in which the points are located, limiting the ability to recognize small patterns and generalize complex scenes. PointNet ++ is a hierarchical neural network that applies PointNet recursively to a set of entry points. Using distances, PointNet ++ can study local features with increasing contextual scale (Fig. 6).
where W -is MLP weights Weighted Summation:
Point Convolution uses spatial-local correlation in data presented densely in grids and provides a basis for studying features from point clouds. One example of such an architecture is PointCNN (Figure 8). Most other semantic networks do not model the necessary relationships between point clouds. RNN-based models will be presented on the example of RSNet (Fig. 9). A key component of the RSNet architecture is a highly efficient module of local dependence between points. RSNet takes as input clouds of not preprocessed points and returns semantic labels for each of them.
Input and output feature blocks are used to generate features independently. In the middle of them, the local dependency module is located. The input function block receives entry points and generates attributes, and the output blocks receive processed input attributes and return final predictions for each point. Both blocks use a sequence of multiple layers to create independent representations of the features for each point. The local dependency module combines an aggregation layer, a bidirectional recurrent neural network (RNN) layer, and a separation layer. The problem of the local context is solved first by projecting disordered points on ordered features and then applying traditional learning algorithms.
Graph-based networks are used to capture the shapes and geometric structures of threedimensional point clouds. First, a point cloud is represented as a set of simple interconnected shapes and super points, then a graph of super points is used to capture the structure and context of information. After that, the large-scale problem of cloud point segmentation is divided into three subtasks: geometrically homogeneous distribution, the embedding of super points, and context segmentation.
One example of a Graph-based architecture is DGCNN (Figure 10). DGCNN is an EdgeConv that is suitable for CNN complex point cloud tasks, including classification and segmentation.
EdgeConv operates on graphs that are dynamically computed at each level of the network. It covers the local geometric structure, preserving the invariance of the permutation. Instead of generating point features directly from their embeddings, EdgeConv generates edge features that describe the relationships between a point and its neighbors. EdgeConv is designed to be invariant to the ordering of neighbors and therefore is an invariant of permutation.
The results of the analyzed methods in the previous section are compared on the datasets S3DIS [15], ScanNet [16], Semantic3D [17], and SemanticKITTI [18]. For this purpose, mean class accuracy (mAcc), overall accuracy (oAcc), and mean class intersection over union (mIoU) metrics are used. The data are taken from the articles of the corresponding algorithms and datasets. S3DIS: all point clouds are obtained without manual intervention using a Matterport scanner. Dataset consists of 271 rooms, which belong to 6 large-scale internal scenes from 3 different buildings, with 6020 sq.m. These areas mainly include offices, training and exhibition spaces, conference rooms.
ScanNet: annotations contain expected calibration parameters, camera positions, threedimensional surface reconstructions, textured grids, dense object-level semantic segmentation, CAD models. The dataset contains annotated RGB-D environment scans. In total, there are 2.5M images in 1513 scans obtained in 707 different locations.
Semantic3D: includes about 4 billion 3D points obtained using static ground-based laser scanners, covering up to 160x240x30 meters of space. Point clouds belong to 8 classes (urban and rural) and contain coordinates, RGB information, and intensity.
SemanticKITTI: an extensive outdoor dataset containing a detailed point annotation of 28 classes. The dataset contains labels for the whole horizontal 360-degree field of view of the rotating laser sensor. Table 3 shows the results achieved in the original articles of methods and datasets. As can be seen from the results presented in this table, the best metrics in the RandLA-Net model are on the Semantic3D and SemanticKITTI datasets, in the PointCNN model -on the S3DIS, ScanNet datasets. However, there are quite a few unknown results on various datasets. Accordingly, we can assume that the RandLA-Net or PointCNN models will work best on the ArCH dataset. However, due to the omitted values, it may turn out that the other models will still be better than the previous two.
The experiments part demonstrates semantic segmentation models' performance on S3DIS, ScanNet, Semantic3D, and SemanticKITTI datasets. For the Semantic3D dataset, the best model is RandLA-Net, which shows high results with oAcc = 94.8 and mIoU = 77.4. In the two previous datasets, the maximum value of oAcc was 85.1 and mIoU 57.26.
The SemanticKITTI dataset is also poorly researched. The mAcc metric is shown for PointNet only and is 29.9. The mIoU metric is presented for PointNet and RandLA-Net and is 17.9 and 53.9, respectively.
Therefore, if we compare the presented methods PointNet, PointNet ++ and RandLA-Net, PointCNN, RSNet, and DGCNN for the S3DIS, ScanNet, Semantic3D, and SemanticKITTI datasets, we can get the following conclusions. Further research is needed on the methods presented on the relevant datasets, as not all possible options have been considered in the official articles in which the models and datasets were first presented. Further research should be performed on the same metrics, finding mAcc, oAcc, and mIoU in all combinations of models and datasets. After the same metrics are received, it will be possible to compare which datasets on which models work best. The last step will be to check the models presented on the ArCH dataset against the same metrics mAcc, oAcc, and mIoU.
The protection of cultural heritage in the urbanization epoch and city development time is critical for preserving history. However, this is quite an enormous task with many risks connected to time, financial, and human resources. Therefore, a solution for automating the monitoring and analysis of data by creating semantic segmentation of point clouds was presented. A risk-informed system based on computational and information technologies will reduce risks and increase the efficiency of using these resources.
The existing solutions were considered, the methods and datasets that correspond to the goal were analyzed, and their results on different metrics were collected and analyzed.
The following steps in continuing this study will be: conducting experiments on the presented methods for the respective datasets; comparison of experimental results on the same metrics; verification of the presented methods on the ArCH dataset.
The study examines ArCH datasets and best techniques for segmenting 3D point clouds: Point-wise MLP with PointNet, PointNet ++ and RandLA-Net, Point Convolution with PointCNN, RNN-based with RSNet, Graph-based with DGCNN. The paper examines the efficiency of the semantic segmentation models PointNet, PointNet ++, RandLA-Net, PointCNN, RSNet, DGCNN on S3DIS, ScanNet, Semantic3D, and SemanticKITTI datasets. The efficiency of semantic segmentation models PointNet, PointNet ++, RandLA-Net, PointCNN, RSNet, DGCNN on S3DIS, ScanNet, Semantic3D, and SemanticKITTI datasets are compared.