This paper describes an approach to managing resources of the supercomputer for e ective execution of stereo photogrammetric tasks with Agisoft PhotoScan software. A research was made to establish the performance characteristics in order to obtain proper deployment. Looking at the PhotoScan calculating process, there are two mandatory steps (aligning photos and building model):
Stereo photogrammetry tasks are a part of a Big Data analyzing and processing task classes. The aim of the photogrammetry is to build 3-dimensional (3D) model based on the set of 2-dimensional (2D) images (photos). Precise modelling of terrain/land environment from the data acquired by drones is extra hard to perform due to huge amount of input data, which can be measured in tens and hundreds of thousands of photos. A supercomputer is needed to e ectively process described task, so that the result will be obtained in a reasonable time.
An approach to managing the supercomputer resources is studied on the
Agisoft PhotoScan Software [
The rst stage is camera alignment. At this stage PhotoScan searches for common points on photographs and matches them, as well as it nds the position of the camera for each picture and re nes camera calibration parameters. As a result a sparse point cloud and a set of camera positions are formed. The sparse point cloud represents the results of photo alignment and will not be directly used in the further 3D model construction procedure (except for the sparse point cloud based reconstruction method). However it can be exported for further usage in external programs. For instance, the sparse point cloud model can be used in a 3D editor as a reference. On the contrary, the set of camera positions is required for further 3D model reconstruction by PhotoScan. 2. Building Dense Cloud.
The next stage is building dense point cloud. Based on the estimated camera positions and pictures themselves a dense point cloud is built by PhotoScan. Dense point cloud may be edited and classi ed prior to export or proceeding to 3D mesh model generation. 3. Next steps depend on the type of model user wants to obtain and have their own speci cs.
In general, PhotoScan reconstructs a 3D model representing the object
surface based on the dense or sparse cloud according to user's choice. There are
two algorithmic methods available in PhotoScan that can be applied to 3D
mesh generation: Height Field { for planar type surfaces and Arbitrary { for
any kind of object [
In this study we are focusing on the obtaining the orthomosaic, therefore these steps will present in the calculation process:
2. Align Photos. 3. Build Dense Cloud. 4. Build DEM (Digital Elevation Model). 5. Build Orthomosaic.
From all stages of processing, building dense cloud is the most resource
demanding [
Known PhotoScan performance studies are conducted for single-machine use case, when all the processing is being executed on the same machine where user works. This is unsuitable for big projects because of inability to process large sets of data. 3
Supercomputer \Polytechnic" is a hybrid complex with peak performance more
than 1 PFlops. Supercomputer was developed by Russian company RSC [
Hybrid supercomputer consists of three clusters: Tornado [
Tornado specs: 668 nodes with two Intel E5-2697 v3 CPUs; 64 GB RAM; IB FDR; 56 nodes also have 2 NVIDIA Tesla K40 GPGPU each. Overall: 1336 CPUs; 18704 x-86 cores; 112 GPGPU; 42752 GB RAM.
Numascale specs: 64 nodes with three AMD Opteron 6380 CPUs; 192 GB
RAM, cache-coherent access to non-uniform memory (CC-NUMA) [
Data Storage System consists of two parts: parallel Lustre Data Storage System (DSS) with 1 PB capacity and modular DSS for cloud with 0.5 PB capacity. 4
Suggested approach is based on the ability of the supercomputer to connect clusters through the Ininiband. By dividing the tasks of analyzing and processing data on several chunks we can combine supercomputer resources in a more e cient way, which results in faster processing.
Application of this approach on Agisoft PhotoScan software can be seen below, with di erent deploy con gurations of supercomputer clusters. Size of input data used for the experiment varies from 10000 photos (100 GB) to 100000 (1 TB). 4.1
10000-50000 Photos Processing Results
2. 8 Tornado nodes. 3. 8 Tornado nodes with GPGPU (Tornado-k40).
In Table 1 you can see the processing times for each step of 10000 photos project with di erent con gurations. The time is shown in minutes.
Tornado and Tornado-k40 di er only in Build Dense Cloud Step, because
GPGPU is only used on this step [
Despite the fact that Numascale nodes have three times more RAM, it does not boost the performance. Numascale nodes are 1.7-4 times slower than Tornado nodes.
Trend holds for projects with size from 10000 to 50000 photos. Obtained performance evaluation will be later used for proper deployment. 4.2
Some problems can be encountered when processing large projects, which contain more than 50000 photos. These issues appear because of the lack of RAM on Tornado nodes, because only 64 GB is available.
Paper [
It was found that during these subtasks Tornado nodes were suspended. Thanks to log analysis routine, the cause of suspend was found. It was happening due to memory over ow.
Also, problematic step and subtask were determined. Problems were occurring in the Align Photos stage. Some experiments were successfully held in order to con rm the theory. To overcome the issue an alteration in cluster con guration was made. Numascale node was added to processing nodes, because it has three times as much memory somewhat comparable performance. Numascale node was included in the nodes list with top priority, so that the server which distributes the tasks will always address the problematic subtasks to the Numascale node.
On the plus side, after adding the Numascale node it became possible to process projects up to 75000 photos.
On the other side, performance slightly dropped because Numascale node is
slower than Tornado and Tornado-k40. Keeping in mind the Amdal's law [
But single Numascale node was not enough to process 100000 photos project. So, four Numascale nodes were uni ed into a group with summary memory of 768 GB. The project was completed successfully, but it was noted that performance of Numascale group is slower than single Numascale node, reducing the performance even more. Results of experiments can be seen in Table 2. The time is shown in minutes.
Memory requirements and processing times can be seen on Fig. 2. From the left to the right (blue colon is processing time in hours, orange { peak memory consumption in GB):
2. 75000 photos project; 40 Tornado-k40 nodes and one Numascale node. 3. 100000 photos project; 40 Tornado-k40 nodes and one Numascale node; No time presented due to fail, there was not enough memory to nish the process. 4. 100000 photos project; 40 Tornado-k40 nodes and four Numascale nodes uni ed into group with shared memory.
The distribution of memory consumption can be seen on Fig. 3. Data acquired from Numascale node. Usually memory consumption is low enough to be run on Tornado nodes, but there are peak(s) that requires a lot more memory, just like on the Fig. 3, where it hits the mark of 120 GB.
Overall performance could be raised by processing several projects at ones. Thanks to the conducted study, following heuristics were made: top priority projects should be placed on Tornado-k40, other projects must run on Tornado. To enhance the stability and be able to process large projects it is better to include one Numascale node for each project, so that most memory demanding subtasks will be run on it. 5
In this paper, an approach was suggested to managing the resources of hybrid supercomputer for maximizing its e ciency and reducing the overall processing time. Results of applying the approach are presented on Agisoft PhotoScan software. PhotoScan performance was analyzed on di erent con gurations and projects. E ective con guration was determined, which can process large projects in reasonable time. It was noted that the same project will run faster on Tornado cluster, rather than on Numascale. However, because using only Tornado nodes it is impossible to process large projects a combined con guration with Numascale node was suggested. Using of shared memory cluster allows overcoming issues with memory over ow by reducing the performance of the system. Without nodes grouping losses aren't that much { only about 10%, but with grouped nodes performance drops signi cantly lower. Also, performance could be boosted by processing several projects simultaneously. Overall optimal con guration is to process several projects at the same time, where each project is computed on Tornado nodes and one Numascale node to avoid memory consumption issues.
Acknowledgments. This work was nancially supported by the Ministry of Education and Science of the Russian Federation in the framework of the Federal Targeted Programme for Research and Development in Priority Areas of Advancement of the Russian Scienti c and Technological Complex for 20142020 (project No. 14.584.21.0022, ID RFMEFI58417X0022) and in the framework of the state assignment No. 2.9517.2017/8.9 (the project theme "Methods and technologies for veri cation and development of software for modeling and calculations using HPC platform with extramassive parallelism").