This paper presents the Multi-view StereoMaker software package. It allows the automatic construction of a stereo image for several objects and accompanying information to them using the multi-view technology. The Multi-view StereoMaker software package is designed to work with arbitrary 3D objects, which can be represented both by the results of calculations of mathematical modeling problems, and by images specified by the user himself. The software package is designed to work with an autostereoscopic monitor. The main purpose of the developed software is to realize the construction of several objects with different stereo bases in one stereo frame. It allows the user to quickly and efficiently build images by varying the stereo base for different objects in order to achieve the most expressive stereo effect. A description of the main functions and capabilities of the first version of the developed system is also presented and the real results of the operation of the Multi-view StereoMaker software complex are presented.
Due to the gradual development and spread of stereo devices in the modern world, there is a need for a more thorough study of the field of stereo-animation technologies and the possibility of their application in scientific research. The construction of stereo-animation representations of scientific research results using such stereo devices opens up many new possibilities for more efficient evaluation of the results obtained.
For example, a three-dimensional representation of the evolution of a process over time helps to verify the computational model and the algorithms used. It also provides the observer with a full understanding of the phenomenon being modeled and popularizes the research and its results for society, including for decision-makers.
With the advent and development of stereo devices, many works have appeared devoted to various
issues of constructing stereo images. Specific problems arising when using a computer system to
generate and visualize a composite multiscreen stereo frame, and methods for solving such problems
are described in sufficient detail in [
2021 Copyright for this paper by its authors. reality systems are described in detail in [17]. The works [21-34] consider various methods for assessing the quality of the stereo effect in the viewer's perception of stereoscopic images.
Stereo imaging is possible using a fairly wide range of different technologies: from classical stereoscopy to depth maps. The choice of technology depends on the type of stereo device available, the field of application, and the immediate objectives of construction the stereo image.
This work is a continuation of the cycle of research works [
The main type of device for which the software system for construction stereo images described in this work is intended is the Dimenco DM654MAS autostereoscopic monitor. An autostereoscopic monitor is capable of displaying an object being rendered in two ways: either using a composite frame containing views of the object being rendered from different angles that form a certain sector of view this method is called multi-view representation, or using depth maps.
When constructing a multi-view stereo image, nine frames forming eight stereopairs ([1 | 2], [2 | 3], [3 | 4],…, [8 | 9]) are combined into one image according to the principle of a 3 × 3 matrix. In the first and last (ninth) frames, the represented object is in its extreme positions. In the first frame, the object is in its extreme right position, in the last frame - in its extreme left. Further, in the middle frames, respectively, from the second to the eighth, at certain equal intervals, it shifts to the left until it reaches the extreme left position. In this case, in each frame, the object is progressively rotated by the same angle and displaced by a given interval. This allows one to achieve the most effective result when construction a stereo image of an object.
The viewer can observe only one of the stereopairs in one position, depending on being in a certain angular sector of observation, but when he starts to move from sector to sector, he receives information about the object using all nine angles, that is, as if looking at the visualization object from different sides.
In papers [
With the advent of such technologies, a number of smaller-scale, but extremely important technical subtasks are coming to the fore, which also need to be solved. One of these tasks is combining images and text information in one frame. Many objects, especially those that represent the results of scientific research, require accompanying information located directly in the same frame as the image. In many cases, the object and the signature, in principle, cannot be separated into different frames, since they constitute a single logical display of information.
In the process of research, analyzed in [
This algorithm is well suited for the implementation of a specific task with a small amount of data that needs to be processed. However, it is only necessary to increase requests for the amount of processed data (the number of images that need accompanying information), and the laboriousness of the implementation of this technology manually becomes obvious, as well as the inefficiency in terms of the amount of labor and time costs.
This process requires a lot of time and a high degree of accuracy and care, since even if it takes place according to a well-established technology, but the result is achieved by using various programs and sequential processing of each frame with setting all the necessary parameters, and therefore it must be automated in order to the user could only set the initial parameters, while the mechanical part was performed by software.
This work is devoted to the development of the task of automating this process and ensuring automatic generation of a frame on which several objects or inscriptions could be placed, each of which has its own stereo base.
The first version of the Multi-view StereoMaker program was registered in 2020 [20]. However, since it has rather limited capabilities for the user, the development of the second version continues to improve its work and add new options for users.
Multi-view StereoMaker is a software package that allows one to create a set of nine images / representations of rotation of three-dimensional objects on a given angular stereobase, and then get them combined into one image.
Multi-view StereoMaker was implemented in Python 3.6. This language was chosen due to such criteria as cross-platform, simplicity of the language and its wide support for additional ready-made functionality. An additional justification was the fact that there were no restrictions on the speed of the software package in the requirements for the program.
The program is designed to work with arbitrary 3D objects with specified parameters. Such 3D objects can be the results of scientific calculations and accompanying information to them, such as accompanying inscriptions, formulas, additional images. Since the visualization of the scientific calculations used was carried out in the Tecplot software package, Multi-view StereoMaker additionally organizes the implementation of interaction with this software package. As a software package for generating arbitrary 3D images, the Blender program was chosen, which is open and free software with wide support for the Python language.
A screenshot of the interface of the Multi-view StereoMaker software package is shown in Figure
Among the main functions of the program, the most important are: selection of an algorithm for generating a stereo image for each object; the ability to preview the generated images according to the specified parameters for each object; setting the parameters of overlaying images on top of each other at user-specified coordinates; the choice of the way to save the final result.
Among the algorithms for generating stereo images, there is a category of "custom images", where the user, respectively, can specify the directory in which his images are located. When choosing the "Tecplot" category, the user must specify a .plt file with the required model and, when setting the necessary parameters, specify the size and width of the output image, the initial rotation angle, the number of rotations, the object rotation angle, and the rotation axis. The last category - the Blender algorithm - involves generating a 3D object either from the original user image in the .svg format, or using internal text generation, as well as specifying additional parameters such as the color and depth of the 3D object, its scale, size and width of the output image, starting angle of rotation, number of turns, axis of rotation, etc.
The Multi-view StereoMaker program consists of the following components, shown in the diagram (Fig. 2). The user must enter the parameters using the provided graphical interface (GUI). The graphical interface allows one to access the generated images preview subsystem for each object. The parameters entered by the user are passed to the data storage subsystem, which creates a directory system for storing the results of the algorithms. Then the parameters are passed to the algorithm generator. Then the finished result is processed and properly formatted.
The algorithm generator generates the appropriate algorithms for the operation of external programs (Tecplot, Blender), after which the obtained algorithms and parameters are transferred to the subsystem for interaction with external programs and data. In the case of transferring custom images, this subsystem will copy the custom images to the resulting directory and transfer them to the image stitching subsystem. In other cases, this subsystem will launch external programs with the algorithms specified for them. Then the image fusion subsystem is called, which performs multiple superposition of a number of images formed for two objects. The result of the program's work is a directory containing algorithms for external programs generated for a specific case, directories with generated images for objects with user-specified parameters, and a directory with the final result.
This program was applied in practice for stereo visualization of real scientific results, and observers comparing the manual version and the generated program recognized that the latter, in terms of clarity and the obtained stereo effect, is not inferior to the manually built version. In [18-19], various images of the results of solving mathematical modeling problems are presented, stereo images for which were generated by Multi-view StereoMaker.
The following figures (Figs. 3,4) show stereo images, constructed from the results of a generalized computational experiment comparing the accuracy of some solvers of the OpenFOAM open source software package. The basic problem for the generalized computational experiment was the classical problem of the formation of a two-dimensional rarefaction wave. This problem has an exact solution. The problem was considered in a parametric formulation with variations in certain ranges of the key parameters of the problem - the Mach number and the angle of deflection of the flow. In Figures 3, 4, the X1 coordinate corresponds to the Mach number, and the Y1 coordinate to the flow deflection angle.
In the calculations, 4 solvers of the OpenFOAM package were considered. The main target functional in this problem was the error of each solver in comparison with the exact solution. The target functional was considered in the ranges of the key parameters, which made it possible to present the error in the form of a function of key parameters and build it in the form of a surface. Figure 3 shows the error surface for the QGDFOAM solver. Figure 4 shows error surfaces for all four solvers that participated in the generalized computational accuracy comparison experiment.
The Multi-view StereoMaker project continues to evolve. It is planned to further develop and improve the program for combining several objects and interactively placing them in a stere=o frame with the simultaneous generation of nine views for each. Additional options will also be developed to fine-tune the generation of stereo images.
The main purpose of this program is the ability for the user to create the widest possible set of stereo image options with varying various parameters in order to visually select the best one.
Compared to the usual flat image, the construction of a stereo image for scientific results allows to increase the insight of the phenomenon and provides an improved sense of the object reality.
The results of constructing a stereo image for an autostereoscopic monitor by the method of multiview representation using the Multi-view StereoMaker software package versions 1.0 and 2.0 are presented. In the course of practical experiments, the technology of automated constructing of multiview stereo images for the results of mathematical modeling problems was developed.
There was also a demonstration of the program's work on specific real problems. Stereo images of the results of a generalized computational experiment on comparative estimation of the accuracy of OpenFOAM solvers for the problem of rarefaction wave are constructed.
In the future, it is planned to further develop custom functions, as well as to solve the problem of interactively placing objects in one stereo frame with the simultaneous generation of nine views for each.
[13] S.V. Andreev, N.A. Bondareva, E.Yu. Denisov, Stereo Presentations Problems of Textual information on an Autostereoscopic Monitor, Scientific Visualization 11 (4) (2019), pp. 90 - 101, doi: 10.26583/sv.11.4.08 [14] S.V. Andreev, A.E. Bondarev, N.A. Bondareva, Stereo images of error surfaces in problems of numerical methods verification, Scientific Visualization, 12 (2) (2020), pp.151-157, doi: 10.26583/sv.12.2.12 [15] Stereo animation "Modeling and visualization of the work of a power plant having complex shape in the stereo animation mode", 2020, URL:https://yadi.sk/i/3CcYhlMB3TCH4r. [16] S.V. Andreev, A.E. Bondarev, N.A. Bondareva, Stereoscopic construction of textual information in presentations of research results on an autostereoscopic monitor, Scientific Visualization 12 (1) (2020), pp.132 - 139, doi: 10.26583/sv.12.1.12 [17] A.D. Zhdanov, D.D. Zhdanov, N.N. Bogdanov, I.S. Potemin, V.A. Galaktionov, and M.I. Sorokin, Discomfort of Visual Perception in Virtual and Mixed Reality Systems, Programming and Computer Software, 45 (4) (2019), pp. 147–155. doi: 10.1134/S036176881904011X [18] S.V. Andreev, A.E. Bondarev, N.A. Bondareva, K.A. Donskoy, V.A. Galaktionov, Automated construction of a stereo image for results of mathematical modeling using the Multi-view StereoMaker software system, Scientific Visualization 12 (5) (2020): pp.165-173, doi: 10.26583/sv.12.5.12 [19] S.V. Andreev, N.A. Bondareva, A.E. Bondarev, Expansion of the Functions of the Multi-View Stereomaker Software for Automatic Construction of Complex Stereo Images, Scientific Visualization 13 (2) (2021), pp. 149 - 156, doi: 10.26583/sv.13.2.10 [20] The program for automatic construction of stereopresentation for an autostereoscopic monitor using the Multi-view method "Multi-View Stereomaker", Patent No. 2020664486. [21] J. Yang, Y. Lin, Z. Gao, Z. Lv, W. Wei, H. Song, Quality Index for Stereoscopic Images by Separately Evaluating Adding and Subtracting, PLoS ONE 10(12) (2015), doi: 10.1371/journal.pone.0145800 [22] J. Zhou, G. Jiang, X. Mao, M. Yu, F. Shao, Z. Peng et al, Subjective quality analyses of stereoscopic images in 3DTV system, in: Visual Communications and Image Processing (VCIP), (2011), pp. 1–4. [23] JS. Lee, L. Goldmann, T. Ebrahimi, Paired comparison-based subjective quality assessment of stereoscopic images, Multimedia tools and applications, 67 (1) (2013), pp.31–48. [24] W.A. Ijsselsteijn, H. de Ridder, J. Vliegen, Subjective evaluation of stereoscopic images: effects of camera parameters and display duration, Circuits and Systems for Video Technology, IEEE Transactions on, 10(2) (2000), pp. 225–233. [25] M. Wöpking, Viewing comfort with stereoscopic pictures: An experimental study on the subjective effects of disparity magnitude and depth of focus, Journal of the society for information display, 3 (3) (1995), pp. 101–103. [26] R. Akhter, ZP. Sazzad, Y. Horita, J. Baltes, No-reference stereoscopic image quality assessment, in: IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 2010, pp. 75240T–75240T. [27] A. Boev, A. Gotchev, K. Egiazarian, A. Aksay, G.B. Aka, Towards compound stereo-video quality metric: a specific encoder-based framework, in: Image Analysis and Interpretation, 2006 IEEE Southwest Symposium on. IEEE, pp. 218–222. [28] J. You, L. Xing, A. Perkis, X. Wang, Perceptual quality assessment for stereoscopic images based on 2D image quality metrics and disparity analysis, in: Proc. of International Workshop on Video Processing and Quality Metrics for Consumer Electronics, Scottsdale, AZ, USA, 2010. [29] M.J. Chen, C.C. Su, D.K. Kwon, L.K. Cormack, A.C. Bovik. Full-reference quality assessment of stereopairs accounting for rivalry, Signal Processing: Image Communication. 28(9) (2013), pp. 1143–1155. [30] Y.H. Lin, J.L. Wu, Quality assessment of stereoscopic 3D image compression by binocular integration behaviors, Image Processing, IEEE Transactions on, 23(4) (2014) pp.1527–1542. [31] F. Qi, T. Jiang, S. Ma, D. Zhao, Quality of experience assessment for stereoscopic images, in:
Circuits and Systems (ISCAS), 2012, IEEE International Symposium on. IEEE, pp.1712–1715. [32] L. Shen, Q. Gao, Objective quality assessment of stereo images with Human Visual characteristic, in: Image and Signal Processing (CISP), 2012, 5th International Congress on. IEEE, pp.1119– 1123. [33] A. Maalouf, M.C. Larabi, CYCLOP: A stereo color image quality assessment metric, in: Acoustics, Speech and Signal Processing (ICASSP), 2011, IEEE International Conference on.
IEEE; pp.1161–1164. [34] K. Ha, M. Kim, A perceptual quality assessment metric using temporal complexity and disparity information for stereoscopic video, in: Image Processing (ICIP), 2011, 18th IEEE International Conference on. IEEE, pp.2525–2528.