Services using omnidirectional images have become increasingly popular. For example, Google Street View enables users to view the scenery of a location online without physically visiting it. However, the use of still images limits the sense of presence. This study proposes a method that focuses on natural elements such as water, sky, and trees within a single omnidirectional image and utilizes deep learning to reproduce their motion in 3D space, generating omnidirectional videos. Experiments demonstrate the efectiveness of the proposed method by comparing results with conventional methods.
a significant amount of time for collecting video data.
To solve this problem, we propose a method that
foVirtual sightseeing services using omnidirectional images cuses on natural objects such as water, sky, and trees
have been increasing. For example, TOWNWARP [
In contrast, Google Street View [
Therefore, for example, if the method [
For these problems, in our previous study [
The flow of the proposed method is as follows. First,
(1) we input an omnidirectional landscape image
containing either sky, water or trees, as shown in figure
1(a). This study assumes that omnidirectional images
are generated by equirectangular projection so that the
bottom pixel is in the direction of gravity obtained from
the accelerometer in the camera. Next, (2) we apply the
semantic segmentation [
Next, (4) we generate the motion of the water surface,
sky and trees by copying pixel values using calculated
optical flows. The motion of the water surface and sky are
calculated by estimating the motion in 3D space based on
the deep learning-based optical flow estimation [
Finally (5) We combine the video of each region gen(a) Example of input image (b) Semantic segmentation
(c) Mask image (d) Inpainting result erated in (4) with the input image using the segmented image as shown in Figure 1(b) to generate a video in which only the water surface, sky, and trees move. Here, alpha blending is performed at the boundary of the mask to reduce the unnaturalness at the boundary between the moving and static regions. The following sections describe the details of motion generation in Step (4). the motion calculated on the plane here only corresponds to a part of the lower part of the omnidirectional image. To handle the entire water area in the omnidirectional image, this study assumes that the motion of the water at any given location is similar. Here, since the projected region from the perspective projection image is a trapezoidal shape, as shown by the red outline in Figure 3, the lfow map of the region is extracted, scaled, interpolated, and shifted to align with the square region projected from the lower half of the omnidirectional image, as shown in Figure 4.
Next, as shown in Figure 2, the , coordinates on the plane obtained by equation (1) are shifted by flow (, ), and projected onto the surface of the sphere. Pixel (2, 2) in the omnidirectional image corresponding to the shifted coordinate ( +, +, ) on the horizontal plane is determined as follows:
For the motion of water, we assume that the water is moving along a planar surface in 3D space. This 3D motion on a plane is represented as a 2D optical flow on the omnidirectional image.
Specifically, we first define the coordinate system for the omnidirectional image and the plane. As shown in Figure 2, position (, , ) on the water surface corresponding to pixel (1, 1) of the omnidirectional image is determined in a coordinate system where the center of the sphere corresponding to the omnidirectional image is at the origin, as follows: ⎡⎤ ⎡ tan ℎ1 cos 2 1 ⎤ ⎣ ⎦ = ⎣ tan ℎ1 sin 2 1 ⎦ ,
where and ℎ are the width and height of the omnidirectional image, and is a negative constant representing the height of the water surface.
In this coordinate system, we compute the flow (, )
at the water surface. First, as shown in Figure 3, a part of
the omnidirectional image is extracted as a perspective
projection image so that its horizon is at the center of
the image height. The optical flow is then estimated by
the deep learning-based method in [
Next, the flow map on the plane is determined from 2 the diferences between the respective projected 3D coordinates. This process is performed on the pixels in the Finally, the diference between the transformed pixel lower half of the perspective projection image. However, (2, 2) and the original pixel (1, 1) is calculated as the (3) ︂[ 2]︂ = [︃ 2 tan− 1 +
+ ℎ cos− 1 √(+)2+( +)2+2 ]︃ . optical flow (, ) on the omnidirectional image. By performing this process for all pixels, the optical flows for the entire water surface on the omnidirectional image are obtained. Based on the optical flows, pixel values are copied to generate an image where the water surface has moved. This process is repeated for each frame, and a video is generated by combining all the frames.
For the motion of the sky, assuming that clouds in the sky
move on a plane in 3D space above the scene, the motion
is estimated in the same manner as for the water. The
optical flows in the upper part of the perspective projection
image in Figure 3 is estimated by the deep learning-based
method [
Note that, as described in section 3.1, by removing all areas other than the sky using inpainting, the plausible sky texture is generated in the areas. Even when the flow is from behind buildings, the generated texture is copied, and the motion of the sky can be reproduced.
For the motion of trees, rather than assuming a single plane like water and sky, we assume that, as shown in Figure 5, they move on a vertical plane perpendicular to the radial line from the center of the sphere to the sphere surface at height 0, for each column. Similar to the sky and water, this 3D motion is expressed as a 2D optical lfow on the omnidirectional image.
Specifically, a reference video is first input and the
optical flows are estimated by Farneback method [
⎣ tan ℎ1 (4) The 3D coordinate shifted on the vertical plane based on the flow map, and the shifted 3D coordinate is reprojected onto the shpere. The flow on the omnidirectinal image is determined by the original and the re-projected pixels. By repeating this process for the number of frames in the reference video, a video with the tree motion is generated.
We conducted experiments to generate a video from a
single omnidirectional image. As input, we used an
image captured with the 360° camera RICOH THETA Z1
and an image obtained from Google Street View, which
were resized to a resolution of 1600 × 800. We used the
image captured with the 360° camera as Case 1, and the
image obtained from Google Street View as Case 2. In the
experiments, we set the height of the planes representing
the sky and water along the Z-axis to 2 and -2,
respectively. We set the focal lengths , and image center
, of the perspective projection image with a
resolution of 384 × 384 to 192. We obtained the motion of the
trees from the reference video as shown in Figure 6. The
generated video consisted of 199 frames. Additionally,
in Case 2, we compared the results with those obtained
by directly applying the conventional method [
4.2.1. Result of Case 1
From these experimental results, we can observe that the sky moves naturally in the sky region, and we can frame) by the proposed method in Case 1. Figures 8 also feel perspective because the clouds just above us and 9 show the result of converting these frames into moves faster than those in the distance. As for the waperspective projection images in a specific direction. In ter, we can see that the water surface moves in various this experiment, we convereted the omnidirectional im- directions, resulting in successfully representing waves. age into the perspective projection image as shown in In the oflw map of the water plane in Figure 10(c), we Figure 10(a). Figure 10(b) shows the the calculated optical can observe various hues between green and yellow, and lfow at the 30th frame. From this flow map, we generated the brightness also varies, indicating that the complex the flow maps of the water and sky planes as shown in motion of the water is well-represented. In contrast, the Figures 10(c) and (d). In these figures, the angle of mo- lfow of the sky shows less variation in hue compared to tion is represented by hue, the relative magnitude of the the water, confirming that it moves in a mostly consistent motion is represented by brightness, and the saturation direction. Regarding the trees, although the motion of is fixed at 1. the trees in the reference video is reflected in the omni
(a) Perspective projection im-(b) Flow of perspective projecage tion image (c) Flow of water plane (d) Flow of sky plane
results. A specific example of this issue is illustrated in Figure 11, where the sun is also considered as part of the sky, making it move in the same way as the clouds. One solution is to extract the sun from the sky by developing a new semantic segmentation method and then to keep the sun in its original position.
Furthermore, while this study focuses on animating natural objects, many tourist spots also have moving man-made objects such as cars and flags. If these objects are not properly animated, the realism of the video is reduced. We should develop a method for animating man-made objects, further enhancing the realism of the video. (a) Omnidirectional image (b) Perspective projection
In this study, we proposed a method for generating videos with motion of natural objects from a single omnidirectional image by the combination of estimating optical lfows using deep learning and considering the motion in 3D space for virtual sightseeing. Through experiments, we confirmed that the proposed method is efective. However, while the water and sky regions moved naturally, the tree regions still show some unnatural motion. In future work, we introduce deep learning for the motion of trees as well.
This research was partially supported by JSPS KAKENHI JP23K21689.