=Paper=
{{Paper
|id=Vol-3297/short2
|storemode=property
|title=Treadmill-framework for Cognitive and Motoric Tests in Mixed Reality Environments
|pdfUrl=https://ceur-ws.org/Vol-3297/short2.pdf
|volume=Vol-3297
|authors=Nils Fischer,Michael Schellenbach
|dblpUrl=https://dblp.org/rec/conf/apmar/FischerS22
}}
==Treadmill-framework for Cognitive and Motoric Tests in Mixed Reality Environments==
https://ceur-ws.org/Vol-3297/short2.pdf
Treadmill-framework for cognitive and motoric tests in
mixed reality environments
Nils Fischer1,* , Michael Schellenbach1
1
Hochschule Ruhr West, Lützowstraße 5, Bottrop, 46236, Germany
Abstract
Combining a traditional treadmill with a virtual reality (VR) environment comes with many obstacles. In order to analyse a
persons gait a laboratory evaluation framework was implemented based on the aforementioned scenario. The actual movement
information is being matched with the virtual movement speed in order to create an immersive walking environment. In
this iteration of the framework a Woodway treadmill was used in combination with the HTC Vive Pro 2 VR headset. The
framework will be used for cognitive and motoric tests which leads to many questions regarding the security aspect and the
overall feeling of uncertainty while walking on a treadmill without being able to see the walkable surface. In order to give the
participants a better understanding of the treadmill we decided to implement a mixed reality approach for this iteration.
Keywords
Mixed Reality, Gait Analysis, Experimental Setup, Walking Environment
1. Introduction interface that allows the control of the treadmill using the
positional information on the treadmill. We expanded
This paper presents a work-in-progress laboratory evalu- upon this idea by using reoccurring API-Calls in order to
ation framework for cognitive and motoric testing in a sync the actual speed with the shown motion within the
secluded environment without any outside disturbances. virtual environment. Using the results of Banton et al.[2]
The goal of this framework is to create a setting, in which we tried to match the virtual environment to reality as
participants can fully focus their attention on walking closely as possible. We wanted to reduce discrepancies
without the fear of getting distracted. In the framework between the two environments wherever possible.
presented in this paper we use the outside cameras of the The research of Fung et al.[3] and Kim et al.[4] already
VR headset to directly show the real treadmill inside the analysed the gait of a person with the combination of a
VR environment. The aim is to analyse the gait of partic- treadmill and a virtual environment. They both specifi-
ipants while they are fully immersed in the simulation. cally analysed the gait of post stroke patients which isn’t
We use different hardware and software components to the targeted group for this framework. For this evalua-
achieve the aforementioned results. tion we target healthy persons in order to see if there are
The main components of the project setup are a tread- irregularities in the gait when walking on the treadmill
mill on which the participants are walking and a virtual while not being able to witness the real world. Moreover,
environment where the movement is being displayed. we want to use this framework to conduct cognitive and
The movement speed of the participants is directly linked motoric tests at the same time. If the safety of the par-
to the walking speed of the character in the virtual envi- ticipant is guaranteed, special circumstances, especially
ronment and the treadmill will accelerate or decelerate therapeutic cases, are a possible future outlook.
according to the participants movements. These infor- The safety of the participants is the highest priority
mation are getting recorded by a camera and submitted when trying to create a virtual walking environment. In
to the treadmill and the VR environment. order to analyse the gait of the participants they have to
create a feeling of security. Tests conducted by Schaefer et
al.[5] show that different kind of groups react differently
2. Related Works to an uncommon walking scenario. Therefore, we revised
Inspired was the idea to use a camera to measure the po- a lot of the components from the first iteration of the
sition of the participants on the treadmill by the work of framework[6] and made it easier to use and understand.
Lichtenstein et al.[1]. They present a feedback-controlled We modified the approach used by Czienskowski et
al.[7] and used an Azure Kinect DK camera in order to
APMAR’22: The 14th Asia-Pacific Workshop on Mixed and Augmented create a self-propelled configuration. This increases the
Reality, Dec. 02–03, 2022, Yokohama, Japan immersion of the framework and allows for a smooth
*
Corresponding author.
transition between the real- and virtual world. Going
$ Nils.Fischer@hs-ruhrwest.de (N. Fischer);
Michael.Schellenbach@hs-ruhrwest.de (M. Schellenbach) forward, we want to conduct tests so discrepancies need
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License to be minimized.
Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
3. Changes from the first iteration treadmill is being controlled based on the positioning on
the treadmill, the participants need to verify themselves
The Oculus Quest virtual reality headset was replaced if they are walking correctly. The most critiqued point
by the HTC Vive Pro 2. This change was made because from the first iteration was the feeling of uneasiness while
of the additional wireless adapter for the HTC Vive Pro walking on a moving object without actually seeing it.
2. The Oculus Quest is a standalone application that With this approach we give all participants the chance
didn’t need a connection to any other device while the to check for themselves if they are walking correctly by
HTC Vive Pro 2 streams a game running on a different simply looking down. This gives the participants a higher
system. A cable is a dangerous aspect while running on sense of security when using the framework.
a treadmill and that’s why the Oculus Quest was chosen
at first. After further testing it became apparent that the
Oculus Quest doesn’t provide the kind of performance
a standard PC can provide. The HTC Vive Pro 2 can
provide better visuals and higher resolutions because the
processing duties are outsourced to a Windows PC.
Another change was made to the game engine. Here
the engine was changed from Unity3D to the Unreal En-
gine 4. This change was made because the reoccurring en-
vironment was a huge problem in the first iteration. With
the use of the Voxel Plugin (https://voxelplugin.com) for
the Unreal Engine 4, it is possible to create highly perfor-
mant landscapes procedurally. With this technology we Figure 1: View of the treadmill inside the environment
can create a road through a forest that doesn’t feel repet-
itive. Furthermore, the underlying landscape provides
the information for which kind of object needs to be ren- In figure 1 the treadmill can be seen while wearing
dered. Therefore, the forest can be rendered on-the-fly the HTC Vive Pro 2 virtual reality headset. The external
which highly reduces the necessary processing power cameras are used to bring the real world into the virtual
and enhances the authenticity of the virtual environment. environment. In this case we use the Reality Mixer appli-
Additionally, we discarded the idea of using controller cation (https://go.hrw.nrw/Eibp97) in order to bring an
inputs for marginal adjustments of the treadmill in order object into the environment based on the position inside
to have a clearer data flow between the different sys- the room. The HTC base stations are used to track the
tems. Moreover, the self-propelled approach gives the headset within the virtual room setup and the objects
participants a higher sense of security because they are within that area can be integrated into any given virtual
controlling the speed of the treadmill directly with their reality application. By looking down while being inside
own movements. the virtual environment we can see the treadmill by using
the position of the treadmill inside the room. This allows
for a safer walking experience overall.
3.1. Mixed Reality Approach
Based on the results of the first iteration of the framework
we decided to abandon the idea of using a motion capture 4. Framework overview
system in order to simulate the real life motions for the
The presented setup is focused around a Woodway PPS55
participants. This was done by streaming the data to
med treadmill which is connected to an Azure Kinect DK.
a virtual character to show the leg movement without
The communication between these two components is
being able to see the actual body. This comes with the
possible due to a Windows PC functioning as the con-
benefit that the participants don’t have to equip the whole
trol unit. The camera records the participants and sends
suit for every session. Moreover, the representation of
the information to the control unit. The position of the
the movement wasn’t accurate all the time when it was
participants on the treadmill can be determined based
being displayed in the virtual environment.
on the distance of the participants to the camera. This
Therefore, we gave up on using a complete virtual
information can then be used to slow down the tread-
reality setup and now chose a mixed reality approach.
mill if the participants are at the back or to increase the
While the participants are fully immersed in the virtual
speed if they are more to the front of the treadmill. This
environment we use the front cameras of the HTC Vice
self-propelled approach will adjust the treadmill speed
Pro 2 to always show the treadmill underneath in order to
according to the walking speed of the participants, which
give the participants a clear view of their position on the
means that the treadmill is completely controlled by the
treadmill. Even though the distance is measured and the
movement of the individuals walking on it.
In order to simulate a secluded environment, the par- a linear, almost flat road in order to provide an environ-
ticipants use a virtual reality headset. In this iteration of ment which resembles the flat surface of the treadmill.
the framework we use the HTC Vive Pro 2 with the Vive There are also no objects on the road so that the partici-
wireless adapter. The virtual reality headset displays a pants don’t feel the need to avoid the obstacle by making
virtual environment created with the Unreal Engine 4 unnatural movements. These would create a disturbance
running on a separate Windows PC. It functions as a for the gait analysis and more importantly could lead to
display while the processing power of the PC is being injuries for the participants.
used for rendering the environment. In order to send the
information to the VR environment we implemented a
web server into the Unreal Project that can change the
movement speed according to the received values.
A separate Raspberry Pi microcomputer is being used
to communicate with the treadmill. It features a ReST-
API that can translate the incoming information from
the Windows PC to hexadecimal values for the treadmill
to understand. This ReST-API will also send the current
speed values to the web server running in the virtual
environment in order to match the actual speed of the
treadmill to the motion in the virtual environment. The
ReST-API will always set a new speed value according
to the distance from the Kinect camera system first and
will send the updated values to the virtual environment
afterwards. This will provide the virtual environment
with incremental updates until the new speed value has
been reached.
The project written in the Unreal Engine 4 features Figure 3: Data flows between the different systems
a large forest that is fully procedurally generated based
on different heights of the underlying landscape. This Figure 3 shows the different systems that are being
means that different kinds of heights will be populated used for the application and how they interact with one
with different kinds of textures and objects. The character
another. The emergency functionality is communicat-
walks along a predetermined road through this forest and ing directly with the treadmill and stopping the whole
encounters different kinds of placements along the way. data exchange process between the treadmill and the
The height map for this simulation is almost flat because Raspberry Pi. The Windows PC running the game and
the simulation is being used for a normal and consistent the web server is only running the game engine and the
gait. However, the design can also be used for simulating HTC Vive Pro 2 is used to display the environment for
slopes if the used treadmill allows for such a feature. the participants. Every new data package starts from
the Azure Kinect DK camera and gets interpreted by the
control unit as the new recommended speed value for
the treadmill. This value is based on the position of the
participants on the treadmill.
The Raspberry Pi is located in the center of the architec-
ture and receives the new values from the Azure Kinect
DK in order to send the new values to both the tread-
mill and the Unreal Project. However, the information is
not being sent to both systems immediately because the
treadmill needs some time to accelerate or decelerate to
the new speed value. Therefore, the information is being
transmitted to the treadmill and afterwards the current
Figure 2: View into the virtual environment speed of the treadmill is used as the new value for the
virtual environment. The Raspberry Pi asks the treadmill
The environment created in the Unreal Engine 4 is for the current speed value on a fixed interval so that
shown in figure 2, where a linear road can be seen. The even the small increments from the last speed value to
participants walk along this road while wearing the vir- the new one are being shown in the virtual environment.
tual reality headset and being on the treadmill. We chose This prevents the participants from seeing a different
speed then they are experiencing on the treadmill itself.
We want to have as few inconsistencies as possible in be an external camera that could be shown inside the
order to reduce the probability of accidents. virtual environment. This would on the one hand break
the immersion a little bit, but would on the other hand
increase the safety of the setup. Here a plethora of op-
tions could be available and the participants could chose
the options they want while conducting the experiment.
We also conducted a first test with a couple of selected
participants in order to get their impressions of the frame-
work in its current form. The overall consensus was that
the experience improved from the first iteration, but they
also gave feedback regarding possible improvements. It
became apparent that the height of the participants was
a huge factor on the overall experience. The larger the
participants, the smaller the angle between the head po-
sition and the front of the treadmill. Higher participants
had to make a larger afford in order to see the front of the
treadmill. Another point was the walking on the tread-
Figure 4: View of the framework’s components mill itself. Here most of the participants needed guidance
in order to find a pleasant movement speed, but most
of the participants could get used to the environment.
In the fourth figure a view of the framework is shown. Therefore, an adjustment period has to be considered
The participant is wearing the HTC Vive Pro 2 while while walking in the environment.
positioned in the middle of the treadmill. In front of
the treadmill is the Azure Kinect DK camera in order to
measure the distance between the participant and the 6. Upcoming Work
camera itself. The Windows PC, from which the Unreal
Engine 4 footage is being streamed, can be seen on the Programmatically the next step is the fine-tuning of the
right hand side. The control unit which translates the software to get the participants positional data using the
distance to a new speed recommendation is shown on the Azure Kinect DK. Walking on the treadmill showed that
left of the camera. The monitor on the right shows the in- it is rather difficult to specifically move to the back of the
game footage, while the laptop shows the camera footage. treadmill in order to slow the treadmill down. Without
This laptop sends the information to the microcomputer any clear indicators where the back end of the treadmill
located in the back of the treadmill. is, it is really dangerous to try to move to the end of
the treadmill. We want to use the camera to stop the
treadmill if a certain limit on the back of the treadmill is
5. Results of technical testing reached. This means that the treadmill should slow down
earlier on the back end then it increases the movement
While walking on the treadmill with the HTC Vive Pro speed while reaching the front.
2 headset with the described features, a couple of future One idea from the previous section was the usage of a
improvements were found. Normally by walking in a multitude of options for the participants to choose from.
straight line humans can just look down by turning the These options could be available to every participant and
eyes without turning the whole head downwards. With they could decide which options are better for their over-
a virtual reality headset a head motion is necessary in all experience. We display the treadmill underneath the
order to see what’s right in front of somebody. This leads virtual character, but other options for orientation on
to the problem we already had with the first iteration the treadmill could be the implementation of positional
that a longer treadmill would make it easier to walk in markers to indicate the position even further. Moreover, a
any virtual environment. With the PPS55 med treadmill slider could help to visualise the movement on the tread-
we don’t see the treadmill comfortably without look- mill in front of the virtual character. Another option
ing down. Showing the treadmill directly improves the could be the installation of an overhead camera to show
overall walking experience significantly, but it can cer- the direct camera footage inside the virtual environment.
tainly be enhanced even further. A visual marker could In order to get a better picture of the needed features we
be shown in front of the participants if the comfortable plan to conduct a test run with a larger group of partici-
walking area was abandoned. This could be a slider that pants after fine-tuning the software. Going forward, this
represents the treadmill as a whole and based on the will help the future development of the framework and
camera position this slider could be updated giving the the experiments we want to use the framework for.
participants another visual support. Another idea would
References
[1] L. Lichtenstein, J. Barabas, E. Peli, R. L. Woods, A
feedback-controlled interface for treadmill locomo-
tion in virtual environments, ACM Transactions
on Applied Perception (2007) 7 – es. doi:10.1145/
1227134.1227141.
[2] T. Banton, F. Durgin, A. Fass, D. Proffitt, J. Stefanucci,
The perception of walking speed in a virtual envi-
ronment, Presence: Teleoperators and Virtual Envi-
ronments (2005) 394 – 406. doi:10.1145/1227134.
1227141.
[3] J. Fung, A. Lamontagne, F. Malouin, B. J. McFadyen,
C. L. Richards, A treadmill and motion coupled
virtual reality system for gait training post-stroke,
CyberPsychology and Behavior 9 (2006) 157 – 162.
doi:10.1089/cpb.2006.9.157.
[4] N. Kim, Y. Kim, B. Lee, W. Min, Effects of virtual
reality treadmill training on community balance con-
fidence and gait in people post-stroke: a randomized
controlled trial, Journal of Experimental Stroke and
Translation Medicine 9 (2016).
[5] S. Schaefer, M. Schellenbach, U. Lindenberger,
M. Woollacott, Walking in high-risk settings: Do
older adults still prioritize gait when distracted by a
cognitive task, Experimental Brain Research (2014).
doi:10.1007/s00221-014-4093-8.
[6] N. Fischer, M. Schellenbach, Simulation of a Vir-
tual Reality Environment for Cognitive and Motoric
Testing with Unity and a Mechanical Treadmill, in:
J. Maiero, M. Weier, D. Zielasko (Eds.), ICAT-EGVE
2021 - International Conference on Artificial Reality
and Telexistence and Eurographics Symposium on
Virtual Environments - Posters and Demos, The Eu-
rographics Association, 2021. doi:10.2312/egve.
20211337.
[7] P. Czienskowski, M. Schellenbach, T. von Oertzen,
Feedback-controlled locomotion in virtual environ-
ments, NordiCHI ’08: Proceedings of the 5th
Nordic conference on Human-computer interaction:
building bridges (2008) 447 – 450. doi:10.1145/
1463160.1463216.