=Paper=
{{Paper
|id=Vol-2485/paper1
|storemode=property
|title=Virtual Environment System for Pirs Space Module Interior
|pdfUrl=https://ceur-ws.org/Vol-2485/paper1.pdf
|volume=Vol-2485
|authors=Andrey Maltsev,Mikhail Mikhaylyuk
}}
==Virtual Environment System for Pirs Space Module Interior==
https://ceur-ws.org/Vol-2485/paper1.pdf
Virtual Environment System for Pirs Space Module Interior
A.V. Maltsev1, M.V. Mikhaylyuk1
avmaltcev@mail.ru|mix@niisi.ras.ru
1
Federal State Institution «Scientific Research Institute for System Analysis of the Russian Academy of Sciences»,
Moscow, Russia
This paper presents original methods and approaches to implement virtual environment system for one of the International Space
Station module. Such system may be used as a part of cosmonaut training complexes in order to orient trainees in the internal space of
the module as well as to develop their skills of onboard equipment control. Proposed system structure consists of three general
components: hardware part, software complex, and 3D scene with virtual model of considered space module. The last two components
are own original products. Our solutions are based on using modern technologies of virtual reality (VR), including VR headset and
Kinect device.
Keywords: virtual reality, simulation, stereo visualization, training complex, real time.
models of hands. After loading the scene, virtual observer is
1. Introduction bound with the operator. An image seen by virtual eyes is
transmitted to the person's eyes with using VR headset. Thanks
One of dynamically developing direction within computer
to several tracking systems (tracking of the operator’s hands and
simulation is creation and adoption of new virtual environment
torso is implemented based on the Kinect, head is tracked by the
systems and simulation-training complexes built on modern
Rift headset), positions and orientations of the observer's
hardware and software bases. They are used to train qualified
elements in virtual environment are synchronized with positions
specialists for controlling complex equipment. Increasingly, a
and orientations of the corresponding parts of the operator’s body
basis of such complexes and system is provided by technologies
in real space. The operator thus gets an opportunity to move and
of virtual reality (VR, [1]).
turn inside virtual interior, turn his head for viewing
Using VR technologies allows carrying out the most fully
environment, and interact with interior elements by means of his
operator immersion in a three-dimensional virtual environment.
hands. For example, buttons of virtual control panel inside space
Furthermore, several significant benefits are achieved compared
module are such elements. Clicking on them causes some actions
to early used approaches without VR. First, there is no need to
(turning on or off the lighting, opening or closing the hatches
create a real model of an environment for each training complex
etc.) defined by control scheme. This scheme is developed when
and system. Implementation and service of such models often
creating virtual scene and saved with it.
require substantial cash investments. Sometimes it's impossible
To implement the system described above, we were
at all. In VR solutions the environment is fully replaced by a
developed original software complex. It consists of three
virtual model displayed to the operator’s eyes through virtual
subsystems, which are responsible for controlling virtual
reality helmets and headsets. Graphic designer creates this model
observer and other elements of virtual environment, computing
by means of 3D modelling system, for example Autodesk 3ds
the dynamics of objects, collision detection and response, and
Max, Autodesk Maya and so on. It's much easier to support and
also virtual environment visualization with per-pixel calculation
update virtual models than real ones. Another advantage of using of realistic lighting. Each of these three subsystems provides
VR technologies in virtual environment systems and training
real-time functionality and is original product, designed from the
complexes is to improve a quality for visual perception by the
ground up without using third-party software components. To
operator of virtual space, therefore bringing it more to a real
achieve our goal, we were also created own virtual scene with
prototype. It allows enhancing the effectiveness of qualified staff Pirs space module interior and model of virtual observer. The
training by means of such systems.
scene was made in Autodesk 3ds Max and converted to a format
VR implementation task is particularly relevant in the field
supported by our software complex. In addition, a control scheme
of space training complexes and systems [2, 4, 5, 7]. This paper
for the virtual control panel was developed by means of our own
presents virtual environment system implementation methods scheme editor. Implementation of hardware part that is
and approaches for an interior of the International Space
responsible for tracking of the operator’s head, body and hands
Station’s (the ISS) one module. Pirs was chosen as such module.
and necessary for rendered image transmission into his eyes is
It is an important element of the Russian Orbital Segment,
based on the Kinect and Oculus Rift devices.
because it includes the docking unit for transportation spacecraft
Figure 1 illustrates general structure of our virtual
of the Soyuz and Progress type, as well as provides a spacewalk
environment system. Consider its components in more detail.
opportunity. Proposed solutions are based on using modern VR
technologies and allow the operator to interact with virtual
nvironment objects in real time. Developed system can be used
to examine the interior details, to orient operators in the
arrangement of important elements (devices, hatches, etc.), and
also to train some skills needed on board the ISS. One of these
tasks is learning to use a control panel placed inside the module.
2. Virtual environment system structure
Virtual environment system proposed by us moves the
operator from real world to virtual interior of Pirs space module.
The working principle of the system is as follows. The user puts
on Oculus Rift headset and enters the workspace of Microsoft
Kinect device. At the time of initialization, the system loads
three-dimensional virtual scene with the interior and a virtual
Fig. 1. Our virtual environment system structure
observer. The observer includes eyes (two virtual cameras) and
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
movements are parallel to vertical, transverse and sagittal body
axes. In addition, another three linear motors and one rotational
are linked with the observer as a whole. They are necessary to
move the observer around the module and to rotate it about
vertical body axis. All motors of the virtual observer are
controlled by software complex with using own control scheme.
2.2. Hardware part
Hardware part of our virtual environment system is based on
Oculus Rift CV1 and Microsoft Kinect devices (fig. 4). The Rift
headset provides transferring stereo images (synthesized by
software complex) of virtual space surrounding the operator to
his eyes. Stereo pair is displayed on LCD screen built into the
Fig. 2. 3D scene of Pirs module interior device. Image separation for left and right eyes is performed by
means of special optics, placed between the screen and the eyes.
Distortions and chromatic aberrations produced by such optics
are compensated with using preliminary correction of displayed
images. This correction is done by the headset software.
Oculus Rift CV1 VR headset also includes tracking system
based on a gyroscope, accelerometer and magnetometer. This
system is used to compute position and orientation of the
operator’s head. Obtained data is sent to software complex that
provides corresponding positions and orientations of virtual
cameras for left and right eyes.
The Kinect device computes three-dimensional coordinates
of user skeleton anchor points in real time based on information
from internal RGB camera and IR sensor. In our work we use
coordinates of wrists and torso points which sent to software
complex for computing virtual hand positions, as well as position
and orientation of virtual observer.
Fig. 3. Virtual control panel
2.3. Software complex
2.1. Virtual scene Software part of developed virtual environment system
Virtual scene created by us and loaded to our system is a consists of original control, dynamics and visualization
detailed virtual model of the International Space Station’s Pirs subsystems. Control subsystem takes as its input a data about
module interior (fig. 2). Except visible elements, it contains pressing buttons of the virtual control panel, as well as current
many bounding volumes (boxes, spheres, cylinders, etc.). They positions and orientations of the operator’s head and body. The
are used for collision detection and response when interactions last arrive from the Kinect device and Oculus Rift CV1 headset.
of the operator with objects are occurred. Also a virtual control Furthermore, control schemes for the observer and the panel are
panel with possibility of pressing its buttons is placed into the loaded from virtual scene. Based on arrived data, the subsystem
scene (fig. 3). Control algorithm for the panel is set by means of computes the schemes and synthesizes control signals which are
special control scheme. Computing this scheme and synthesis of sent to dynamics subsystem.
control signals are done in software complex. Dynamics subsystem computes new positions and
To integrate the operator with virtual environment described orientations of virtual cameras, hands and other objects of the
above, a virtual observer is added to it. This observer consists of scene, based on signals received from control subsystem. It also
two virtual cameras (for the operator’s left and right eyes) and performs detection and response of collisions between virtual
hands. The tasks of current virtual environment system version hands and the scene objects including buttons of the virtual
are visual inspection of Pirs module interior by the operator and control panel. Computed data is sent to visualization subsystem
learning the basics of control with using the panel buttons. implemented by us with using the OpenGL graphics library,
Therefore, all fingers of virtual hands except the index fingers GLSL shader technology and CUDA. It performs distributed
are bent. The observer's movements and turns are provided by rendering of virtual environment by means of multi-core
means of virtual motors (linear and rotational). Our software graphics processor (GPU). Visualization subsystem result is a
complex controls these motors in accordance with information stereo pair in the form of a single image separated into two equal
arrived from the Kinect and Oculus Rift devices about the parts. This image is sent to the Oculus Rift CV1. It also is
operator’s position and motions in real world. To implement duplicated on standard means for displaying (a monitor or a
rotate and tilt of the operator’s head, there are two rotational projector).
motors hierarchical linked with pair of cameras. Three linear
motors are set in every virtual hand. It provides that their 3. Implementation results
Developed methods and approaches were implemented in the
form of virtual environment system prototype for Pirs space
module interior. The complex consists of high performance
computing unit based on Intel Core i7-8700K processor and
NVIDIA RTX 2080 graphics card, Oculus Rift CV1 headset and
its sensor, Microsoft Kinect v2.0 device, display. Software
modules were realized with using object-oriented language C++,
Fig. 4. Oculus Rift CV1 and Microsoft Kinect devices
They provide nine degrees of freedom (9DOF) for each virtual
hand. An orientation of real hand wrist is computed based on data
received from a gyroscope, accelerometer and magnetometer.
Finger movements are controlled by two sensors set on each of
them.
An important problem in creating virtual environment
systems for cosmonaut training is also a simulation of
weightlessness conditions. Using a basin in the hydro lab or
special suspension system can be considered as variant of its
solution. However, both options have their advantages and
disadvantages, requiring serious and depth study in future
researches.
5. Conclusions
At this paper we propose methods and approaches for
creating virtual environment system based on modern virtual
reality technologies. Using stereo headset provides high level of
immersing the operator in virtual environment. Proposed
solution may be adapted for other models of the ISS modules.
Fig. 5. Immersing the operator in virtual interior As a next step, it is planned to improve our virtual
of Pirs space module environment system to implement realistic interaction of the
operator’s hands with space module interior objects. To solve
the OpenGL 3D graphics library, shader language GLSL 4.3 and this problem, it is expected to develop new methods and
the CUDA parallel computing architecture. Stereo visualization of algorithms based on using VR gloves.
high polygonal virtual model of Pirs space module interior is performed
in real time. The frame rate is about 300-400 FPS. 6. Acknowledgements
Figure 5 illustrates an example of applying developed
complex to immerse the user in virtual environment. The The publication was made within the state task on carrying
operator feels that he is inside Pirs space module, sees its interior out basic scientific research (GP 14) on the topic (project) “34.9.
and virtual hands. User presses the control panel buttons by Virtual environment systems: technologies, methods and
means of virtual index fingers, which movements is a copy of his algorithms of mathematical modeling and visualization” (0065-
wrists’ movements. 2019-0012).
4. Future work 7. References
Unfortunately, using the Kinect device doesn’t provide full [1] Boletsis C. The New Era of Virtual Reality Locomotion: A
interaction of the operator’s hands with virtual objects. In this Systematic Literature Review of Techniques and a Proposed
work it was used as the initial stage of creating virtual Typology. Multimodal Technologies and Interaction,
environment for Pirs space module. A promising evolution 1(4):24, 2017.
direction for this and other similar systems is using specialized [2] Masalkin A.I., Torgashev M.A. Experience of Using the
devices with sensors to track the operator’s hands. Today, the Simulation Systems of Visual Environment in the
most efficient solutions among them are Oculus Touch Simulators of Manned Space Vehicles. Piloted Flights in
controllers and virtual reality gloves (fig. 6). Space, 2:36-42, 2015.
The Touch device looks like a joystick, but has a specially [3] Perret J., Vander Poorten E. Touching Virtual Reality: A
designed ergonomic shape. Its construction includes an Review of Haptic Gloves. In Proceedings of 16th International
accelerometer and IR sensor which allow computing the position Confernce on New Actuators, pages 270-274, 2018.
and orientation of the operator’s wrist with high accuracy. In [4] Sergeev A. V., Gook M. Yu. Mobile Space Robot Control
addition, the controller tracks when the thumb and index fingers with Use of Virtual Reality. Piloted Flights in Space, 4: 44-
squeeze, as well as the palm state (squeezed or relaxed). 52, 2018.
Important feature of the Touch is haptic feedback which allows [5] How NASA uses virtual reality to train astronauts.
real hand to feel a contact with virtual environment objects. It is https://spacecenter.org/how-nasa-uses-virtual-reality-to-
implemented by means of vibrating motor. train-astronauts, 2018.
Optimal way to realize full interaction between hands and
virtual objects is modern VR gloves [3]. Unlike the Touch they [6] Manus VR: World's leading VR Gloves for training.
provide a control of all hand fingers and allow programing the https://manus-vr.com.
feedback with more flexibility (there is a possibility of [7] Stand of training the crews of the international space station
independent impact on individual hand areas). For example, the using virtual reality elements.
Manus VR Gloves [6] are one of such virtual reality devices. http://www.gctc.ru/main.php?id=135.
Fig. 6. Manus VR gloves and Oculus Touch controller