Earthquakes are natural disasters that we cannot avoid in our daily lives in some countries. Therefore, it is important to raise awareness of disaster prevention and prepare for earthquakes. One of the methods to raise awareness of disaster prevention is to use earthquake simulators. For example, mobile earthquake simulators reproduce actual shaking using a car equipped with a vibration device, and VR-based earthquake simulators enable users to experience earthquakes in a virtual space. However, such earthquake simulators that use special and imaginary environments are unrealistic. In this study, we propose a method to make users feel more realistic than the conventional earthquake simulators by applying substitutional reality. The proposed method gives shaking to the video of the user's current location and make real objects fall over using augmented reality.
the environment has many pieces of actual furniture, it is necessary to draw the background image when the Earthquakes are natural disasters that we cannot avoid furniture is moved virtually. One option is to apply diin our daily lives in some countries. In addition, large minished reality (DR) technology [6, 7, 8, 9] to remove earthquakes often threaten our lives. For this reason, it is the furniture and draw the background image, but it is important to raise awareness of disaster prevention and still dificult for the existing DR methods to remove the prepare for earthquakes on a daily basis. variously arranged furniture in high quality for real-time
One of the methods to raise awareness of disaster pre- video. vention is to use earthquake simulators. For example, mo- In this study, we propose an earthquake simulator that bile earthquake simulators using a vehicle reproduce the allows users to experience realistic earthquakes in the shaking that mimics an actual earthquake, and VR-based location where they are by applying substitutional realearthquake simulators [1, 2, 3] enable users to experience ity (SR) [10] with AR. This is also a concrete example of earthquakes in a virtual space by using a head mounted applying the idea in the literature [11]. Specifically, the display (HMD). However, with the mobile earthquake proposed simulator presents the user with an omnidirecsimulators, the experience is limited to special environ- tional image captured in the past at the user’s current ments such as cars, and earthquakes can only be experi- location to make the user believe that the video is the enced at special places such as events. In addition, while current scene. The simulator then shakes the video and earthquake simulators using VR can be used anytime and makes real objects fall over in the video. Since the proanywhere using an HMD, they lack a sense of reality posed method uses a past omnidirectional image as the because they use imaginary environments and objects scene of the current location, it is possible to prepare in VR space. Therefore, it is questionable whether they in advance the background image of the furniture that actually increase disaster prevention awareness. moves due to the earthquake by inpainting.
To address this issue, an earthquake simulator using augmented reality (AR) that allows users to experience earthquakes in the location where they are has been pro- 2. Proposed earthquake simulator posed [4, 5]. In this method, to allow users to experience earthquakes using a video see-through HMD, the video In this section, we explain the proposed earthquake simuimages are shaken, and virtual objects such as furniture lator using substitutional reality. In the following, we first that are synthesized on the video images are moved or describe the generation of a virtual space that mimics a fallen over. However, the method basically assumes that real scene, and the method for producing an earthquake. the room for the experience has almost no furniture. If
In this study, when producing an earthquake, we not only generate the shaking of video, but also cause objects to fall over. To do this, we place virtual objects that mimic real objects, which actually exist at the place, in a virtual space. For example, to make a cabinet fall over, a rectangular object is generated in the virtual space, and a photograph of the actual cabinet is mapped onto it. The object generated in this way is placed so that they overlap with the actual object in the field of view from the center of the sphere, as shown in Figure 3.
In addition, if we just place the virtualized real object, the object in the background remains in the view after it has fallen over. Therefore, we remove the object from the omnidirectional image, as shown in Figure 4. It is preferable to use inpainting [12, 13, 14, 15] to automatically remove the objects from the omnidirectional image, but if the quality of the inpainting result is not good in the target environment, we can remove it manually using a paint tool, etc.
We simulate an earthquake by shaking the sphere and the virtualized real objects, and then make the virtualized real object fall over towards the user. Specifically, we associate the virtualized real object in Figure 3 with the sphere in Figure 2, and shake them together. Note that, the behaviors of the sphere and the virtualized real object are not exactly the same because we use random values obtained using Perlin noise. In addition, since a large earthquake is composed of an initial tremor and a main tremor, we prepare two diferent magnitudes of shaking.
Next, we make the object fall over. Here, to make the user feel more fear of earthquakes and raise their awareness of disaster prevention, we make the object fall over towards the user. To do this, as shown in Figure 5, we place a small ball and a tilted plane, which the user cannot see during the experience, behind the virtualized real object. The small ball is rolled and hit against the object by an operation or a pre-set time, and the virtualized real object falls over towards the user through physical calculations. At this time, to prevent the virtualized real object that has fallen over from falling under the floor, a transparent plate with a thickness is placed in the sphere.
To enhance the realism of the earthquake, we introduce sounds in addition to visual information. Here, two kinds of sounds of buildings creaking due to shaking are played at the time of small shaking and at the time of large shaking, respectively, corresponding to the magnitude of the shaking. We also add a sound for collisions with the lfoor after the object has fallen over. A rectangular object invisible to the user is placed at the floor position, and the sound is produced when the virtualized real object collides with the rectangular object.
To raise awareness of earthquake disaster prevention in the university laboratory, where the participants spend a lot of time, we conducted the experiment in a situation where an earthquake occurs while the participants are in the laboratory. First, as shown in Figure 6, the participants are asked to wear an HMD and sit on a chair placed 3. Experiments in the same position where the omnidirectional image was captured. Next, the user is asked to look around 3.1. Overview while viewing the omnidirectional image to get familiar with the view shown in Figure 7. After that, they exIn this experiment, we asked participants to experi- perience a small tremor representing the initial tremor, ence the developed earthquake simulator and evaluate followed by a large tremor representing the main tremor. whether it raises their awareness of disaster prevention. When the large tremor occurs, the mechanism for making Ten male students in their twenties in Faculty of Infor- the virtualized real object fall over is activated, and the mation Science and Technology, Osaka Institute of Tech- virtualized real object falls over toward the participant nology, participated in the experiment. We used RICOH as shown in Figure 8. After the experience, we asked the THETA Z1 to capture the omnidirectional image, and participants to answer the five questions, as shown in HTC VIVE Pro2 as the HMD. We used Unity (2019.4.29f1) Table 2, on a 5-point Likert scale: 1 (strongly agree), 2 to develop the proposed simulator, and PC specifications (somewhat agree), 3 (neither agree nor disagree), 4 (somefor the experiment are shown in Table 1. The experimen- what disagree), and 5 (strongly disagree). We also asked tal scene is shown in Figures 1 to 5, and the background them to write comments freely. image of the falling object shown in Figure 4 was created manually using a paint software. 3.3. Results and discussion the way the cabinet fell over looked like a light, weightless fall. In addition, some said that even though they were sitting in a position where the cabinet could not reach them even if it fell over, the cabinet seemed to be close enough to hit them in the virtual space, which made them feel like it was not realistic.
Regarding Q4, Approximately half of the participants agreed and half disagreed with feeling fear. Some of the comments from those who agreed are that he could recognize the danger of objects falling over, and that the shaking was greater than he had expected, which reminded him of the fear of earthquakes. On the other hand, some participants said that they felt less fear because only one object fell, and others said that if all the objects in the scene could be reproduced virtually, it would increase the sense of reality and fear. From these comments, we confirmed that we could create a sense of danger and fear by making the virtualized real object fall over towards the users during a large earthquake. However, the effect was limited because the content was not suficiently developed this time.
Finally, regarding Q5, many participants thought that the proposed system increase the disaster prevention awareness. From this result, we think that it is efective to virtually experience an earthquake in the place where we actually spend a lot of time with the HMD using augmented substitutional reality.
Note that, one of the participants commented that if a system prepared multiple patterns of things falling over in advance and instructed people to respond according to the pattern, it would be more efective at increasing awareness of disaster prevention. From this comment, we consider that it would be more efective to raise awareness of disaster prevention by not just having people experience an earthquake, but also by having them practice what to do when an earthquake occurs.
The results of the questionnaire for five items are shown In this study, we proposed a earthquake simulator that in Figure 9. We discuss the proposed simulator based on makes users feel more realistic by applying substitutional the results of the five questions. reality. The proposed simulator provides a shaking of
Regarding Q1, since many participants agreed, we con- the view image, which was captured at the user’s current sider that we could reproduce the virtualized real scene location, and falling over of a cabinet, which is a virtualthat were close to the actual scene. ized reality object. In future work, we will use the HMD’s
Regarding Q2, we consider that many people agreed camera to switch between real-time and past images, as with the reality of the earthquake because the shaking in the original substitutioal reality [10], to create a more was divided into two parts: the initial tremor and the realistic earthquake experience. main tremor. However, since we created only one virtualized real object that was shaking, one of the participants Acknowledgments commented that the shaking did not feel very realistic.
Regarding Q3, while many people also felt that the This research was partially supported by JSPS KAKENHI cabinet falling over was realistic, more people disagreed JP23K21689. than for the previous questions. We think this is because ronments via omnidirectional video, in: Proceedings of IEEE International Symposium on Mixed [1] C. Li, W. Liang, C. Quigley, Y. Zhao, L.-F. Yu, Earth- and Augmented Reality Adjunct (ISMAR-Adjunct), quake safety training through virtual drills, IEEE 2018, pp. 222–225.
Transactions on Visualization and Computer Graph- [12] J. Jam, C. Kendrick, K. Walker, V. Drouard, J. G.ics 23 (2017) 1275–1284. S. Hsu, M. H. Yap, A comprehensive review [2] R. Suzuki, R. Iitoi, Y. Qiu, K. Iwata, Y. Satoh, Ai- of past and present image inpainting methods, based vr earthquake simulator, in: Proceedings Computer Vision and Image Understanding 203 of Virtual, Augmented and Mixed Reality: Appli- (2021) 103147. URL: https://www.sciencedirect.com/ cations in Health, Cultural Heritage, and Industry, science/article/pii/S1077314220301661. 2018, pp. 213–222. [13] N. Kawai, T. Sato, N. Yokoya, Image inpainting [3] A. Sudiarno, A. D. W. P, Analysis of human factors considering brightness change and spatial locality and workloads in earthquake disaster evacuation of textures and its evaluation, in: Proceedings of simulations using virtual reality technology, IOP Pacific-Rim Symposium on Image and Video TechConference Series: Materials Science and Engineer- nology, 2009, pp. 271–282.
ing 1003 (2020) 012082. [14] N. Kawai, N. Yokoya, Image inpainting considering [4] S. Chotchaicharin, J. Schirm, N. Isoyama, symmetric patterns, in: Proceedings of InternaH. Uchiyama, K. Kiyokawa, Compelling ar tional Conference on Pattern Recognition, 2012, pp. earthquake simulation with ar screen shaking, 2744–2747. in: Proceedings of IEEE International Sympo- [15] T. Tanaka, N. Kawai, Y. Nakashima, T. Sato, sium on Mixed and Augmented Reality Adjunct N. Yokoya, Iterative applications of image com(ISMAR-Adjunct), 2021, pp. 298–299. pletion with cnn-based failure detection, Journal of [5] S. Chotchaicharin, J. Schirm, N. Isoyama, D. V. Mon- Visual Communication and Image Representation teiro, H. Uchiyama, N. Sakata, K. Kiyokawa, Com- 55 (2018) 56–66. pelling ar earthquake simulation with ar screen shaking, in: Proceedings of International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments, 2021. [6] J. Herling, W. Broll, High-quality real-time video inpainting with pixmix, IEEE Transactions on Visualization and Computer Graphics 20 (2014) 866–879. [7] N. Kawai, T. Sato, N. Yokoya, Diminished reality based on image inpainting considering background geometry, IEEE Transactions on Visualization and
Computer Graphics 22 (2016) 1236–1247. [8] S. Mori, J. Herling, W. Broll, N. Kawai, H. Saito,
D. Schmalstieg, D. Kalkofen, 3d pixmix: Image inpainting in 3d environments, in: Proceedings of IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2018, pp. 1–2. [9] M. Kari, T. Grosse-Puppendahl, L. F. Coelho, A. R.
Fender, D. Bethge, R. Schutte, C. Holz, Transformr: Pose-aware object substitution for composing alternate mixed realities, in: Proceedings of IEEE International Symposium on Mixed and Augmented
Reality, 2021, pp. 69–79. [10] K. Suzuki, S. Wakisaka, N. Fujii, Substitutional reality system: A novel experimental platform for experiencing alternative reality, Scientific Reports 2 (2012). [11] S. Okeda, H. Takehara, N. Kawai, N. Sakata, T. Sato,
T. Tanaka, K. Kiyokawa, Toward more believable vr by smooth transition between real and virtual envi