In augmented reality (AR) games, markers are often used to synthesize and present virtual objects on camera images. However, if the AR markers remain visible on the images, it reduces the quality of the AR experience. This problem can be solved by using diminished reality (DR), which visually removes real objects from real-time images and presents the background. However, conventional methods of diminished reality have dificulties with presenting natural background images in areas containing local luminance changes such as specular reflections. In this study, we propose a method for AR marker hiding that can handle specular reflections by using 3D Gaussian Splatting (3DGS), which generates free viewpoint images from a set of 2D images. The proposed method synthesizes a texture with appropriate specular reflections in the marker area according to the current camera pose in AR. Experiments demonstrate the efectiveness of the proposed method by comparing results with conventional methods.
1. Introduction
reproduced in the AR marker area according to the
camera pose. In the process of performing AR, the current
Games using augmented reality (AR) provide realistic camera pose is estimated from the marker, and the image
visual experiences by synthesizing and presenting 3D vir- is rendered using the camera pose in the scene generated
tual objects in camera images, and markers are often used by 3DGS, and the rendered image is synthesized into
to estimate camera poses and determine the positions of the marker area to achieve marker hiding with specular
virtual objects in such games. However, if the AR mark- reflection. In the following sections, we first provide an
ers remain visible when 3D virtual objects are displayed, overview of conventional methods and then describe the
the quality of the AR experience is reduced. This problem proposed method.
can be solved using diminished reality (DR) [
In this study, we focus on specular reflections whose ously captured background image, and the third method
positions change according to camera movement and uses inpainting to estimate the background texture based
propose a novel method of diminished reality to improve on the information around the area.
the quality of background images synthesized on AR In the method using multiple cameras at diferent
viewmarkers placed on planes where specular reflections oc- points [
Methods using inpainting [
As for the photometric adjustment, while many of the losses of the AR marker areas are excluded, so that the
latter methods adjust the luminance of the background 3D scene is reconstructed as if the AR markers in the
image for global luminance changes [
The proposed method falls into a hybrid area between 3D Gaussian Splatting and the details of the proposed the second category using images captured in advance, method are described. and the third category using inpainting. However, unlike the conventional methods in the second category, the 3.2. Overview of 3D Gaussian Splatting images captured in advance includes the target marker 3D Gaussian Splatting (3DGS) reconstructs a 3D scene to be removed, and it is not necessary to capture the represented by a 3D Gaussian distribution from a 2D background behind the marker. Also, unlike the conven- image set through training and can generate images from tional methods in the third category, instead of removing new viewpoints that are not included in the 2D image the target object from the image using patch-based or set via volume rendering. Before training, COLMAP is deep learning-based inpainting, we construct a 3D scene used to estimate the intrinsic and extrinsic parameters of without the target marker by manipulating the loss dur- the cameras, as well as the 3D coordinates of the feature ing 3DGS training. These enable to achieve diminished points for each image in the 2D image set. 3DGS uses the reality that reproduces appropriate specular reflections 2D image set along with the estimated camera poses and in the AR marker area according to the camera pose. the 3D coordinate of each feature point as training data. In the training of 3DGS, the parameters of the 3D Gaus3. Proposed method sian function representing the Gaussian distribution in 3D space are initialized, and the loss between an image 3.1. Overview selected at random from a set of 2D images and the rendered image with the same camera pose is calculated by The proposed method is divided into two processes: pre- the diference in pixel values at each pixel and the DSSIM. process and process during the AR experience. The The Gaussian distribution representing the 3D scene is lfowchart of the entire proposed method is shown in Fig- updated according to the gradient calculated beased on ure 1. In the pre-process stage of the proposed method, the loss.
(b) Rendered image
In the pre-processing stage, many images are captured from various camera poses, ensuring that AR markers are included in the images. These images are used as input for the training of 3D Gaussian Splatting. In this study, the training is conducted in such a way that no loss occurs in the AR marker area, allowing free viewpoint images to be generated as if the AR markers were not present in the scene.
Specifically, the four corners of the AR marker in the input image are obtained, and the inner area containing these corners and its surroundings are filled with a fixed pixel value, as shown in Figure 2(a). The same area is iflled with the same pixel value in the rendered image, as shown in Figure 2(b), to reduce the loss in the AR marker area to zero. This results in reconstructing the 3D scene as if the AR markers did not exist. In this case, as shown in Figure 3, in the space constructed by 3D Gaussian Splatting, a virtual light source is generated at a position symmetrical to the actual light source with respect to the plane on which the marker exists, based on the reflection of specular light. By rendering an image in this constructed environment according to the camera pose, specular reflections appear in the marker area at appropriate positions.
In the process during the AR experience mentioned below, the AR marker in the real-time video is detected, and the camera pose is calculated in a coordinate system where the center of the AR marker is the origin (marker coordinate system). Based on this information, a virtual camera is set up and the same view is rendered in the reconstructed 3D scene. However, since the coordinate system of the 3D scene (COLMAP coordinate system) is established during processing by COLMAP and difers from the marker coordinate system, simply using the current camera pose as the virtual camera one in the reconstructed 3D scene does not yield the same view with the camera image.
Therefore, before training with 3D Gaussian Splatting, the COLMAP coordinate system is transformed to the marker coordinate system using the model aligner, which is one of the functions of COLMAP. Specifically, the poses of all cameras are transformed so that all camera positions in the COLMAP coordinate system match the camera positions in the marker coordinate system, which are obtained by detecting the AR marker. Examples of the coordinate system and camera poses before and after the transformation are shown in Figures 4(a) and 4(b), respectively. (a) Before adjustment (b) After adjustment
(a) Certain camera pose
(b) Diferent camera pose In the process during the AR experience, an image is rendered according to the current camera pose and synthesized into the AR marker area of the real-time video as positions of specular reflections around the marker are the background image. However, if the rendered image consistent between the images to be compared, the lumiis copied directly, the diference in intensity of images nance of the local specular reflection does not afect the between the camera used for training data collection global luminance adjustment. in 3DGS and the AR camera may cause the inconsistency between the texture of the AR marker area and its 4. Experiments surrounding texture in the real-time video as shown in Figure 5(a). To demonstrate the efectiveness of the proposed method,
To eliminate the inconsistency caused by the difer- comparative experiments were conducted between the
ence in intensity between the textures, we employ the proposed method and the conventional method [11] in a
intensity adjustment methods in [
Note that, while the conventional methods [
Figure 10 shows the rendered results obtained by original 3DGS training and the proposed 3DGS training, which excludes the AR marker area. These results demonstrate (a) Virtual light source in rendered image when virtual camera is placed under the floor and facing downward (b) Rendered image when virtual camera is fac
ing upward from below the floor that the method excluding the marker areas successfully generates a floor without the marker. Figure 12: Constructed environment seen from the side.
Figure 11 shows rendered images when the virtual camera is placed under the floor, and facing downward and upward. From the figure, we confirmed that a vir- conventional method [11], the results of the proposed tual light source was generated not at the actual light method before and after intensity adjustment. From source position but under the floor, allowing the light to the experimental results, we can see that the proposed reach the viewpoint and appear as a specular reflection. method properly represents specular reflection in the Figure 12 shows the constructed environment from the area inside the marker, while the specular reflection is disside. From the figure, we can see that the floor in the connected at the boundary of the marker by the conven3D scene are represented not as a solid plane but as a tional method. In addition, the intensity adjustment of hierarchical arrangement of Gaussian distributions with the proposed method achieves adequate global luminance information on the color and opacity of the objects. For adjustment. Figure 14 shows the results from another this reason, not only specular reflections, but also the viewpoint. We can confirm that the shape of specular colors of the floor in areas other than specular reflections reflection is slightly dificult to recognize, but compared are reproduced in the marker area. to the result of the conventional method, the specular
Next, we show experimental results during AR experi- reflection can be definitely observed. The frame rate of ence. Figure 13 shows the input image, the result of the the proposed method was 8-10 fps.
(b) Conventional method (c) Before intensity adjust-(d) After intensity adjustment
ment (Proposed method)
In the experiments conducted on black marble-like tiles, 121 images were used as training data. Figure 15 shows an example of the training data. Figures 16 shows the ren- produced by the conventional method, are more clearly dered results obtained by the original 3DGS training and visible in the result of the proposed method than that in the proposed 3DGS training. In this scene, the proposed Experiment 1. Although the proposed method produces method also makes the marker invisible, and black tiles some artifacts in the results, it can generate specular appear on the marker area. However, the fineness of the reflections that are geometrically consistent in the AR white texture is slightly diferent from the surrounding marker area. The frame rate of the proposed method was texture. 8-10 fps in this experiment as well.
Figures 17 and 18 show the input images, the results of the conventional method [11], the results of the proposed method before and after intensity adjustment during the AR experience with diferent viewpoints. In this experiment, the specular reflections, which were not re(c) Before brightness adjust-(d) Proposed method(After ment brightness adjustment)
In the experiments conducted with a transparent board on the floor, 122 images were used as training data. Figure 19 shows an example of the training data. Figures 20 shows the rendered results obtained by original 3DGS training and the proposed 3DGS training. In this scene as well, we can confirm that the markers have been naturally erased from the scene.
Figures 21 and 22 show the input images, the results of the conventional method, the results of the proposed (a) Input image method before and after intensity adjustment. In this experiment, the conventional method [11] could not reproduce specular reflection as well as the other experiments. On the other hand, the proposed method rendered the specular reflections. We confirmed that the geometrical consistency of the specular reflections in the AR marker area with the proposed method is better than that observed in Experiment 1. However, we also confirmed the diference in luminance between the specular reflections in the AR marker area and the surrounding area. In this experiment, the frame rate of the proposed method was 8-10 fps.
From the results mentioned above, we confirmed that the proposed method is able to represent specular reflections at geometrically appropriate locations, which could not be reproduced by the conventional method. However, for planes with fine textures, as in Experiment 2, we found it dificult to reproduce the similar fine texture on the marker. In addition, as shown in Experiment 3, we confirmed that the intensity adjustment by the proposed method is efective for global diference in intensity in areas other than specular reflection areas, but it is not sufifcient for adjusting the brightness of specular reflections. We consider that this is because the intensity adjustment is performed in the same way for the entire marker area, regardless of the presence or absence of specular reflection. Therefore, one solution to this problem may be to separate the intensity adjustment method for specular reflection areas and other areas.