Combining a traditional treadmill with a virtual reality (VR) environment comes with many obstacles. In order to analyse a persons gait a laboratory evaluation framework was implemented based on the aforementioned scenario. The actual movement information is being matched with the virtual movement speed in order to create an immersive walking environment. In this iteration of the framework a Woodway treadmill was used in combination with the HTC Vive Pro 2 VR headset. The framework will be used for cognitive and motoric tests which leads to many questions regarding the security aspect and the overall feeling of uncertainty while walking on a treadmill without being able to see the walkable surface. In order to give the participants a better understanding of the treadmill we decided to implement a mixed reality approach for this iteration.
interface that allows the control of the treadmill using the
positional information on the treadmill. We expanded
This paper presents a work-in-progress laboratory evalu- upon this idea by using reoccurring API-Calls in order to
ation framework for cognitive and motoric testing in a sync the actual speed with the shown motion within the
secluded environment without any outside disturbances. virtual environment. Using the results of Banton et al.[
The main components of the project setup are a tread- irregularities in the gait when walking on the treadmill
mill on which the participants are walking and a virtual while not being able to witness the real world. Moreover,
environment where the movement is being displayed. we want to use this framework to conduct cognitive and
The movement speed of the participants is directly linked motoric tests at the same time. If the safety of the
parto the walking speed of the character in the virtual envi- ticipant is guaranteed, special circumstances, especially
ronment and the treadmill will accelerate or decelerate therapeutic cases, are a possible future outlook.
according to the participants movements. These infor- The safety of the participants is the highest priority
mation are getting recorded by a camera and submitted when trying to create a virtual walking environment. In
to the treadmill and the VR environment. order to analyse the gait of the participants they have to
create a feeling of security. Tests conducted by Schaefer et
2. Related Works al.[
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)
by the HTC Vive Pro 2. This change was made because of the additional wireless adapter for the HTC Vive Pro 2. The Oculus Quest is a standalone application that didn’t need a connection to any other device while the HTC Vive Pro 2 streams a game running on a diferent system. A cable is a dangerous aspect while running on a treadmill and that’s why the Oculus Quest was chosen at first. After further testing it became apparent that the Oculus Quest doesn’t provide the kind of performance a standard PC can provide. The HTC Vive Pro 2 can provide better visuals and higher resolutions because the processing duties are outsourced to a Windows PC.
Another change was made to the game engine. Here the engine was changed from Unity3D to the Unreal Engine 4. This change was made because the reoccurring environment was a huge problem in the first iteration. With the use of the Voxel Plugin (https://voxelplugin.com) for the Unreal Engine 4, it is possible to create highly performant landscapes procedurally. With this technology we can create a road through a forest that doesn’t feel repetitive. Furthermore, the underlying landscape provides the information for which kind of object needs to be rendered. Therefore, the forest can be rendered on-the-fly which highly reduces the necessary processing power and enhances the authenticity of the virtual environment.
Additionally, we discarded the idea of using controller inputs for marginal adjustments of the treadmill in order to have a clearer data flow between the diferent systems. Moreover, the self-propelled approach gives the participants a higher sense of security because they are controlling the speed of the treadmill directly with their own movements. 3.1. Mixed Reality Approach Based on the results of the first iteration of the framework we decided to abandon the idea of using a motion capture system in order to simulate the real life motions for the participants. This was done by streaming the data to a virtual character to show the leg movement without being able to see the actual body. This comes with the benefit that the participants don’t have to equip the whole suit for every session. Moreover, the representation of the movement wasn’t accurate all the time when it was being displayed in the virtual environment.
Therefore, we gave up on using a complete virtual reality setup and now chose a mixed reality approach. While the participants are fully immersed in the virtual environment we use the front cameras of the HTC Vice Pro 2 to always show the treadmill underneath in order to give the participants a clear view of their position on the treadmill. Even though the distance is measured and the treadmill is being controlled based on the positioning on the treadmill, the participants need to verify themselves if they are walking correctly. The most critiqued point from the first iteration was the feeling of uneasiness while walking on a moving object without actually seeing it. With this approach we give all participants the chance to check for themselves if they are walking correctly by simply looking down. This gives the participants a higher sense of security when using the framework. In figure 1 the treadmill can be seen while wearing the HTC Vive Pro 2 virtual reality headset. The external cameras are used to bring the real world into the virtual environment. In this case we use the Reality Mixer application (https://go.hrw.nrw/Eibp97) in order to bring an object into the environment based on the position inside the room. The HTC base stations are used to track the headset within the virtual room setup and the objects within that area can be integrated into any given virtual reality application. By looking down while being inside the virtual environment we can see the treadmill by using the position of the treadmill inside the room. This allows for a safer walking experience overall.
med treadmill which is connected to an Azure Kinect DK.
The communication between these two components is possible due to a Windows PC functioning as the control unit. The camera records the participants and sends the information to the control unit. The position of the participants on the treadmill can be determined based on the distance of the participants to the camera. This information can then be used to slow down the treadmill if the participants are at the back or to increase the speed if they are more to the front of the treadmill. This self-propelled approach will adjust the treadmill speed according to the walking speed of the participants, which means that the treadmill is completely controlled by the movement of the individuals walking on it.
In order to simulate a secluded environment, the par- a linear, almost flat road in order to provide an environticipants use a virtual reality headset. In this iteration of ment which resembles the flat surface of the treadmill. the framework we use the HTC Vive Pro 2 with the Vive There are also no objects on the road so that the particiwireless adapter. The virtual reality headset displays a pants don’t feel the need to avoid the obstacle by making virtual environment created with the Unreal Engine 4 unnatural movements. These would create a disturbance running on a separate Windows PC. It functions as a for the gait analysis and more importantly could lead to display while the processing power of the PC is being injuries for the participants. used for rendering the environment. In order to send the information to the VR environment we implemented a web server into the Unreal Project that can change the movement speed according to the received values.
A separate Raspberry Pi microcomputer is being used to communicate with the treadmill. It features a ReSTAPI that can translate the incoming information from the Windows PC to hexadecimal values for the treadmill to understand. This ReST-API will also send the current speed values to the web server running in the virtual environment in order to match the actual speed of the treadmill to the motion in the virtual environment. The ReST-API will always set a new speed value according to the distance from the Kinect camera system first and will send the updated values to the virtual environment afterwards. This will provide the virtual environment with incremental updates until the new speed value has been reached.
The project written in the Unreal Engine 4 features Figure 3: Data flows between the diferent systems a large forest that is fully procedurally generated based on diferent heights of the underlying landscape. This means that diferent kinds of heights will be populated with diferent kinds of textures and objects. The character walks along a predetermined road through this forest and encounters diferent kinds of placements along the way.
The height map for this simulation is almost flat because the simulation is being used for a normal and consistent gait. However, the design can also be used for simulating slopes if the used treadmill allows for such a feature.
Figure 3 shows the diferent systems that are being used for the application and how they interact with one another. The emergency functionality is communicating directly with the treadmill and stopping the whole data exchange process between the treadmill and the Raspberry Pi. The Windows PC running the game and the web server is only running the game engine and the HTC Vive Pro 2 is used to display the environment for the participants. Every new data package starts from the Azure Kinect DK camera and gets interpreted by the control unit as the new recommended speed value for the treadmill. This value is based on the position of the participants on the treadmill.
The Raspberry Pi is located in the center of the architecture and receives the new values from the Azure Kinect DK in order to send the new values to both the treadmill and the Unreal Project. However, the information is not being sent to both systems immediately because the treadmill needs some time to accelerate or decelerate to the new speed value. Therefore, the information is being transmitted to the treadmill and afterwards the current speed of the treadmill is used as the new value for the virtual environment. The Raspberry Pi asks the treadmill for the current speed value on a fixed interval so that even the small increments from the last speed value to the new one are being shown in the virtual environment.
This prevents the participants from seeing a diferent speed then they are experiencing on the treadmill itself.
be an external camera that could be shown inside the virtual environment. This would on the one hand break the immersion a little bit, but would on the other hand increase the safety of the setup. Here a plethora of options could be available and the participants could chose the options they want while conducting the experiment.
We also conducted a first test with a couple of selected participants in order to get their impressions of the framework in its current form. The overall consensus was that the experience improved from the first iteration, but they also gave feedback regarding possible improvements. It became apparent that the height of the participants was a huge factor on the overall experience. The larger the participants, the smaller the angle between the head position and the front of the treadmill. Higher participants had to make a larger aford in order to see the front of the treadmill. Another point was the walking on the treadmill itself. Here most of the participants needed guidance in order to find a pleasant movement speed, but most of the participants could get used to the environment. Therefore, an adjustment period has to be considered while walking in the environment.
While walking on the treadmill with the HTC Vive Pro 2 headset with the described features, a couple of future improvements were found. Normally by walking in a straight line humans can just look down by turning the eyes without turning the whole head downwards. With a virtual reality headset a head motion is necessary in order to see what’s right in front of somebody. This leads to the problem we already had with the first iteration that a longer treadmill would make it easier to walk in any virtual environment. With the PPS55 med treadmill we don’t see the treadmill comfortably without looking down. Showing the treadmill directly improves the overall walking experience significantly, but it can certainly be enhanced even further. A visual marker could be shown in front of the participants if the comfortable walking area was abandoned. This could be a slider that represents the treadmill as a whole and based on the camera position this slider could be updated giving the participants another visual support. Another idea would
The participant is wearing the HTC Vive Pro 2 while positioned in the middle of the treadmill. In front of the treadmill is the Azure Kinect DK camera in order to measure the distance between the participant and the 6. Upcoming Work camera itself. The Windows PC, from which the Unreal Engine 4 footage is being streamed, can be seen on the Programmatically the next step is the fine-tuning of the right hand side. The control unit which translates the software to get the participants positional data using the distance to a new speed recommendation is shown on the Azure Kinect DK. Walking on the treadmill showed that left of the camera. The monitor on the right shows the in- it is rather dificult to specifically move to the back of the game footage, while the laptop shows the camera footage. treadmill in order to slow the treadmill down. Without This laptop sends the information to the microcomputer any clear indicators where the back end of the treadmill located in the back of the treadmill. is, it is really dangerous to try to move to the end of the treadmill. We want to use the camera to stop the treadmill if a certain limit on the back of the treadmill is 5. Results of technical testing reached. This means that the treadmill should slow down earlier on the back end then it increases the movement speed while reaching the front.
One idea from the previous section was the usage of a multitude of options for the participants to choose from.
These options could be available to every participant and they could decide which options are better for their overall experience. We display the treadmill underneath the virtual character, but other options for orientation on the treadmill could be the implementation of positional markers to indicate the position even further. Moreover, a slider could help to visualise the movement on the treadmill in front of the virtual character. Another option could be the installation of an overhead camera to show the direct camera footage inside the virtual environment.
In order to get a better picture of the needed features we plan to conduct a test run with a larger group of participants after fine-tuning the software. Going forward, this will help the future development of the framework and the experiments we want to use the framework for.