Dynamic Self-Avatar Motion Retargeting in Virtual Reality Alessandro Clocchiatti 0 Computer Science Department, University of Torino , Corso Svizzera 185, 10149 Torino , Italy 55 60

In Virtual Reality (VR), users can feel embodied through a virtual body that mimics their physical one in terms of location, behavior, and movement. Several studies have explored users' perception of self-avatar movement retargeting. However, the boundaries of dynamic changes of user movements and the process of adaption to new self-avatar behaviors remain unclear and require additional investigations. Exploring this motion retargeting techniques is crucial to both enhance the availability of this technology and improve its applications in rehabilitation therapies.

 eol>Virtual Reality Virtual Embodiment Visuomotor Illusion Motion Retargeting Post-stroke Rehabilitation 1. Introduction

One of the primary advantages of Virtual Reality (VR) is its ability to enable users to explore virtual environments and interact with digital objects from a first-p erson perspective, providing an immersive experience that evokes both psychological and physical sensations of being present in the virtual environment. Many systems provide users with a virtual body, called self-avatar, that is placed, behaves, and moves like the physical body [ 1, 2 ], inducing the so-called Sense of Embodiment (SoE). Several users may have restricted physical spaces or experience motor disabilities that reduce their movement capabilities [ 3 ]. In response, various techniques have been developed to alter users’ movement, aiming to enhance their overall experience within the virtual environment [ 4, 3 ]. Nevertheless, it is crucial to investigate how users can adapt to these new self-avatar movements without compromising both SoE and the accuracy of their interactions.

In this context, the investigation of motion retargeting techniques becomes crucial to ensure the widespread accessibility of VR technology. By refining and implementing motion retargeting approaches, VR experiences can be tailored to the individual needs and capabilities of users, making virtual reality more inclusive and immersive for all.

Having an embodied self-avatar makes VR a possible alternative to traditional rehabilitation therapies (See Palacios-Navarro et al. for a review [ 5 ]).

Typical virtual reality rehabilitation therapies on upper and lower limbs rely on approaches such as Constraint-induced movement therapy (CIMT) in which the induced motion is provided by an external robotic tool [ 6 ], and mirror-therapy exercises, in which afe cted limb movements are simulated by mirroring the healthy limb ones[ 7 ].

Post-stroke patients can be also treated using a setup made of a screen showing a pre-recorded movie of a moving hand, placed over the real patient’s arm (Kinesthetic Illusion Induced by Visual Stimulation, or KiNVIS) [ 8 ]. With this approach, the seen body movements are difer ent from the physical ones, creating an illusory perception of kinesthesia and improving individual motor function.

The possibility of retargeting self-avatar movements makes this technology to be a valid option for rehabilitation therapy based on induced motion illusion. However, according to the authors’ knowledge, this rehabilitation approach in Virtual Reality is still novel. The efe ctiveness of this rehabilitation approach in virtual reality may depend on the ability of the technology to accurately reproduce and alter user’s movements through a virtual body, hiding the real physical body movements. In this scenario, it is important to understand how much difer ence between virtual and physical body maximize the induced movements without afe cting SoE.

Additionally, introducing a gradual variation in the gap between the virtual and real body, rather than maintaining a consistent level of alteration throughout the entire exercise, could prove efe ctive and potentially enhance rehabilitation outcomes.

 2. Related works

In recent years, several works focused on the alteration of the movements of the self-avatar and the user’s perception of the virtual body behavior. Soccini et al. [ 9, 10 ] define d the induced ifnger movements efe ct as the involuntary physical hand motion induced by the sight of the alien movement (alien motion) of the self-avatar fingers, spotted only when SoE is present. Gonzales Franco [ 11 ] define d the self-avatar follower efe ct as the involuntary modification of the physical movements in order to be similar to the virtual body ones. In particular, the study shows how the introduction of an alteration in self-avatar movements makes users follow their self-avatar itself, resulting in a drift. The results suggest that participants show a stronger tendency to follow the avatar when the alteration of movements is introduced gradually, rather than instantaneously. Furthermore, gradual introduction of movement modifications leads to a higher Sense of Embodiment compared to instantaneous alterations.

Similarly, Burin et al. [ 12 ] develop an immersive virtual reality experiment in which user’s were asked to repeat a specific movement (draw continuous straight vertical lines). The results showed that when it is introduced a gap between the actual movements of the user and the self-avatar movements, the motor performance is attracted towards the embodied virtual body movements. The alteration of the self-avatar movements was used to support the learning of specific hand positions and movements.

While these works primarily focus on understanding how the alteration of self-avatar movements afe cts the SoE and real user movements, other studies investigated the user adaptation to the modifie d behavior of the virtual body.

Soccini et al. [ 13 ] investigated users’ adaptation to frequent and abrupt changes in self-avatar movements. The results suggest that users are able to adapt to the new self-avatar behavior without nullifying the Sense of Embodiment, even when the alterations occur frequently over time.

Lilija et al. [ 14 ] proposed a technique in VR in which the position and pose of the user’s hand is corrected to be closer to the target movement. The alteration introduces a mismatch between the actual hand of the user and its VR representation. Despite the results suggesting that the correction of the self-avatar movements improved the short-term retention of the trained movements and the motor learning, the optimal amount of alteration of the self-avatar movements and the duration of the exercise has not been clarifie d. This approach could hold significant potential in the field of rehabilitation, as it ofers opportunities for targeted motor training and skill acquisition. By fine-tuning self-avatar movements, it may be possible to improve motor function and coordination in individuals undergoing rehabilitation programs.

In all the works presented, the alteration of the self-avatar movements does not cancel the Sense of Embodiment. Despite that, it is not clear how users can adapt to the new self-avatar movements such as more complex virtual interactions.

 3. Project description

In a Virtual Reality system, a self-avatar is considered by the users as their own body, and it is important to further investigate the limits of the SoE and its components, such as Sense of Ownership and Sense of Agency. Furthermore, understanding human perception and physical reactions to dynamically retargeted self-avatar movements, which difer from those of the physical body, is crucial.

So far, we already investigated user’s perception to the frequent changes of movement remapping and their ability to adapt to the altered self-avatar behavior [ 13 ]. In this study, participants were asked to fle x/extend their left arm towards two targets while the self-avatar movements dynamically changed (Figure 1). The goal was to understand how users adapt to these frequent changes. The analysis of the results confirms that Sense of Embodiment persists despite the introduction of movement alterations. Furthermore, results suggest that the subjects get better results of adaptation when the self-avatar behavior changes frequently.

Our initial findings indicate that users can quickly adapt to new self-avatar retargeted movements. However, in order to consider this technique as a viable option for future movement and interaction alternatives, it is important to further investigate how motion alteration and users adaptation to these new behaviors are linked. In order to achieve this goal, it is important to ifrst answer the following questions:

Q1: Do users adapt better to gradual self-avatar motion retargeting compared to abrupt alteration without afe cting embodiment? Q2: How does the gradual or abrupt motion retargeting influence the adaptation and the interaction in the virtual environment? How does it afe ct movements and interaction accuracy? Q3: Is it possible to enhance the movement and interaction area without compromising both Sense of Embodiment and accuracy? Can users adapt to a new virtual space temporarily or permanently?

To address these questions, we will explore them based on the experimental design proposed by Soccini et al. [ 13 ], and will include difer ent retargeting patterns and interactions with objects. Specifically , the movement tasks used in the previous study will be replaced with interaction-based tasks, such as whac-a-mole or grab interaction.

These future directions hold potential for application in VR rehabilitation scenarios, where self-avatars serve as guides for users. The gap between virtual body movements and physical movements determines the potential for induced movements [ 15 ]. Furthermore, the application of KiNVIS in Virtual Reality is still novel, and further exploration of self-avatar motion retargeting could contribute to the advancement of this rehabilitation approach. Firstly, in current rehabilitation therapies based on the induced motion illusion, the degree of alteration between virtual and physical body is stable and does not change throughout the sessions [ 8 ]. However, the variation of the self-avatar behavior during the rehabilitation process may be more efe ctive. Secondly, there are still no clear criteria to determine the optimal level of self-avatar movement alteration that can maximize the exercise outcome and rehabilitation efe cts. Additionally, it’s important to understand how the self-avatar behavior should vary in order to increase the induction of motion without afe cting the Sense of Embodiment.

4. Conclusion

Thanks to the rapid growth and improvement of digital technologies, virtual reality systems and applications are used in several difer ent areas, including medical and therapeutic fields. While in the last few years several studies regarding Virtual Reality interaction and VR rehabilitation therapies have been carried out, there is still much work to do in order to make these systems widely available and more efe ctive. Latest developments in VR ofer the opportunity to provide users with virtual bodies, and understanding how self-representation and user’s kinetic reaction to altered self-avatar movements work becomes essential to give a solid contribution in the evolution of Virtual Reality technology.

While my Ph.D. project primarily focuses on post-stroke rehabilitation, the study of selfavatar motion remapping is important to facilitate interaction in VR to a broader range of people. In particular, the dynamic retargeting of the self-avatar movements could be analyzed in various aspects such as interaction with the virtual environment or locomotion. For example, gradually exaggerating the user’s movements could enable interaction and movement within a larger virtual space than the physical space available, without afe cting the SoE and the interaction eficacy .

 [1] K. Kilteni , R. Groten , M. Slater , The sense of embodiment in virtual reality , Presence: Teleoperators and Virtual Environments 21 ( 2012 ) 373 - 387 . URL: https://doi.org/10.1162/ pres_a_00124. doi: 10 .1162/pres_a_ 00124 . [2] M. Slater , B. Spanlang , M. V. Sanchez-Vives , O. Blanke , First person experience of body transfer in virtual reality , PloS one 5 ( 2010 ) e10564 . [3] B. A. Cohn , A. Maselli , E. Ofek, M. Gonzalez-Franco , Snapmove: Movement projection mapping in virtual reality , in: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR) , IEEE, 2020 , pp. 74 - 81 . [4] I. Poupyrev , M. Billinghurst , S. Weghorst , T. Ichikawa, The go-go interaction technique: non-linear mapping for direct manipulation in vr , in: Proceedings of the 9th annual ACM symposium on User interface software and technology , 1996 , pp. 79 - 80 . [5] G. Palacios-Navarro , N. Hogan , Head-mounted display-based therapies for adults poststroke: A systematic review and meta-analysis , Sensors 21 ( 2021 ) 1111 . URL: https: //doi.org/10.3390/s21041111. doi: 10 .3390/s21041111. [6] A. L. Borstad , R. Crawfis , K. Phillips , L. P. Lowes , D. Maung , R. McPherson , A. Siles , L. Worthen-Chaudhari , L. V. Gauthier , In-home delivery of constraint-induced movement therapy via virtual reality gaming , Journal of patient-centered research and reviews 5 ( 2018 ) 6 . [7] D. B. Mekbib , D. K. Debeli , L. Zhang , S. Fang, Y. Shao , W. Yang , J. Han, H. Jiang , J. Zhu , Z. Zhao , et al., A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke , Annals of the New York Academy of Sciences 1493 ( 2021 ) 75 - 89 . [8] M. Okawada , F. Kaneko , K. Shindo , M. Yoneta , K. Sakai , K. Okuyama , K. Akaboshi , M. Liu, Kinesthetic illusion induced by visual stimulation influences sensorimotor event-related desynchronization in stroke patients with severe upper-limb paralysis: A pilot study , Restorative Neurology and Neuroscience 38 ( 2021 ) 455 - 465 . URL: https://doi.org/10.3233/ rnn-201030. doi: 10 .3233/rnn-201030. [9] A. M. Soccini , M. Grangetto , T. Inamura , S. Shimada , Virtual hand illusion: The alien finger motion experiment , in: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) , IEEE, 2019 . URL: https://doi.org/10.1109/vr. 2019 . 8798193 . doi: 10 .1109/vr. 2019 . 8798193 . [10] A. M. Soccini , The induced finger movements efe ct , in: SIGGRAPH Asia 2020 Posters, ACM , 2020 . URL: https://doi.org/10.1145/3415264.3425448. doi: 10 .1145/ 3415264.3425448. [11] M. Gonzalez-Franco , B. Cohn , E. Ofek , D. Burin , A. Maselli , The self-avatar follower efe ct in virtual reality , in: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) , IEEE, 2020 . URL: https://doi.org/10.1109/vr46266. 2020 . 00019 . doi: 10 .1109/vr46266. 2020 . 00019 . [12] D. Burin , K. Kilteni , M. Rabufetti , M. Slater , L. Pia, Body ownership increases the interference between observed and executed movements , PLOS ONE 14 ( 2019 ) e0209899 . URL: https://doi.org/10.1371/journal.pone.0209899. doi: 10 .1371/journal.pone. 0209899 . [13] A. M. Soccini , A. Clocchiatti , T. Inamura, Efe cts of frequent changes in extended selfavatar movements on adaptation performance , Journal of Robotics and Mechatronics 34 ( 2022 ) 756 - 766 . [14] K. Lilija , S. Kyllingsbaek , K. Hornbaek , Correction of avatar hand movements supports learning of a motor skill, in: 2021 IEEE Virtual Reality and 3D User Interfaces (VR) , IEEE, 2021 . URL: https://doi.org/10.1109/vr50410. 2021 . 00069 . doi: 10 .1109/vr50410. 2021 . 00069 . [15] A. M. Soccini , F. Cena , The ethics of rehabilitation in virtual reality: the role of self-avatars and deep learning , in: 2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR) , IEEE, 2021 , pp. 324 - 328 .