This work is devoted to the problem of restoring realistic rendering for augmented and mixed reality systems. Finding the light sources and restoring the correct distribution of scene brightness is one of the key parameters that allows to solve the problem of correct interaction between the virtual and real worlds. With the advent of such datasets as, "LARGE-SCALE RGB + D," it became possible to train neural networks to recognize the depth map of images, which is a key requirement for working with the environment in real time. Additionally, in this work, convolutional neural networks were trained on the synthesized dataset with realistic lighting. The results of the proposed methods are presented, the accuracy of restoring the position of the light sources is estimated, and the visual difference between the image of the scene with the original light sources and the same scene. The speed allows it to be used in real-time AR/VR systems.
Augmented and mixed reality systems are being used in many tasks, however the incorrect illumination of the virtual world objects may cause discomfort in the perception of the reality, in which objects of the real and virtual worlds are mixed and as a result this limits the time that a person can be in the mixed reality, and further it restricts the practical use of the mixed reality systems in various areas, for example - in education or training.
This article is devoted to convolutional neural network methods (CNN) for solving the global scientific problem in the field of the physically correct and effective restoration of illumination conditions and optical properties of real-world objects during the synthesis of images of the virtual world.
First, this work is focused on determining the real power of illumination of the light flux and its position in an environment. For this a manually synthesized sample of images with realistic optical parameters of the medium were used. Although the sample consists of only 260 images (221 were used for training, and39 for the test), the neural network at the output classifies with good accuracy the real optical parameters of the illumination of the medium. These are usually divided by the strength of illumination into 5 classes, where the first is 0 lumens, which means it is not lit at all, while grade 5 is the source of illumination of an ordinary room lamp.
Moreover, for the reconstruction of the depth of an environment the "Large-Scale RGB + D Dataset" was used, which was obtained using Kinect v2 and Zed stereo camera and their disparity maps. 2
The idea of environment segmentation is already trivial, but every year new approaches,
neural network architectures and datasets appear that improve the previous results. An
analysis of outdoor lighting using a fully convolution network is presented in [
Works [
The “LARGE-SCALE RGB+D DATABASE” dataset contains synchronized RGBD
frames from both Kinect v2 and Zed stereo camera. For the outdoor scene, they first
generated disparity maps using an accurate stereo matching method and converted them
using calibration parameters. A per-pixel confidence map of disparity is also provided.
The scenes are captured at various places, e.g., offices, rooms, dormitory, exhibition
center, street, road etc., from Yonsei University and Ewha University. This dataset has
been used to train convolutional neural networks in projects [
Segmentation of Illuminated Areas of Light Using CNN and Large-Scale RGB+D Dataset 3
[
To train a neural network for classification depth maps a sorted dataset "Large-Scale RGB + D Dataset" of indoor images was used, which consists of 1609 images for training and 503 images for testing. These images were obtained using Kinect v2 and Zed stereo cameras, with the calculation of their disparity maps and restoration of the pixel confidence. Sorted "Large-Scale RGB + D Dataset" comes with high-resolution and low-resolution quality, in current work was used the low-resolution pack with squeezing all images to 224*224 pixels, so that it would be possible to train them on the architecture of VGG 16-Net.
The working distance of the Kinect is up to 6 meters. The level of gradation between the distances of measurement levels is about ~30 cm. In total, in this approach with the neural network of the VGG 16-Net architecture 20 output neurons were used, to produce distance measurement with a 20-level heat map. As a convolution network architecture, it was decided to use the VGG16 Net architecture because it was successfully used in many classification tasks, it consists of 5 blocks with convolution, pooling, and “ReLU” activation function between layers and the optimization method is “Nesterov”.
For lighting classification method was used manually synthesized dataset of lighting of different rooms. The task of a fully convolutional network is to classify each pixel of an image into one class. That is, passing through all convolutional layers, the network characterizes a certain area of the image to one class in accordance with the power of illumination.
The architecture is presented on figure 1. The neural network training for the classification of light took 50 epochs, for depth map - 200 epochs. The training took place on a GeForse GTX 1080Ti video card. The original image was fed with 5 light area masks and 20 depth data masks. Dataset images are shown in figures 2 and 3.
Fig. 3. The training dataset images for depth classification.
Segmentation of Illuminated Areas of Light Using CNN and Large-Scale RGB+D Dataset 5 The history of training and results of work are presented in figures 4 and 5. In this work, neural networks were trained to restore light sources and depth maps of the indoor scene. The architecture of this network is good for classifying data with many classes. The speed allows it to be used in real-time systems. The recognition accuracy of light sources in some scenes turned out to be quite good; in the future, it is planned to improve the coordinate recovery algorithm for greater accuracy, which can easily restore the distance to any point in the image, but working with images significantly faster than with 3D models. 5
In further work it is planned to improve the results by using other neural network architectures and image processing using computer vision algorithms to strengthen the output results.