We experience, understand, and interact with the world through the body [ 9 ]. Especially with the rise of AR and VR, interactive technologies o er new possibilities for physical engagement. Users of immersive technologies can manipulate virtual objects, traverse through ctional landscapes, and interact with the VE through increasingly more complex and creative means. The actions of the immersed user are often expressed through interacting with speci c virtual objects or the context of the activity, generating an importance of understanding the body in relation to the VE. Current Cross-Reality research includes sharing a eld of view among multiple users across the Reality-Virtuality continuum [ 1, 12 ], tracking the positions of external users in a VE perceived by VR users [ 11 ], or synchronising the manipulation of select objects in the VE between di erent Cross-Reality users [ 4 ]. However, despite the extensive theoretical research in Cross-Reality Interaction, few focus on the impact of body language in interaction design. 2

Systems exist to make VR users aware of what is happening in their surroundings. Noti VR [ 3 ], for instance, explores di erent noti cations and interruptions. These interruptions can be physical, such a person or a pet, or digital, such as a text message or voice call. The VR Motion Tracker [ 13 ] presents a widget that enables VR users to track external persons. In RealityCheck [ 7 ] a system is presented that composites images from the real world into the VE. RealityCheck also enables external users to view the VE via a projection into the physical environment.

Projections can be used in di erent ways to visualise the VE to external users. ShareVR [ 5 ] presents a way to support interaction between VR and external users via a oor projection and a tracked display. The tracked display functions as a "window into the VE". The oor projection shows the spatial layout of the VE, projected by two projectors set up in the environment. An alternative to placing the projector in the environment is to make it head-mounted. A head-mounted projector has been explored for VR [ 14 ] and AR [ 8 ]. In both cases a small motor controlled projector was positioned on top of the user’s head to project virtual content into the environment.

There are di erent ways of using an external screen to visualise the VE. One way is to mount one or more small screens onto the HMD [ 1, 6 ]. Tablets have also been used to enable collaboration between a VR user and non-immersed user. Grandi et al. [ 4 ] performed a study investigating collaboration of a VR and tablet AR user. They found that VR-AR asymmetric collaboration performed better than AR-AR collaboration but worse than VR-VR collaboration. TransceiVR [ 10 ] enables communication between an immersed VR user and an external tablet user.

Vishnu [ 2 ] is a system for a remote expert to assist a local agent in a maintenance procedure. The remote expert can perform the correct actions in VR, which are then shown to the local agent via AR. The local agent can then perform the correct actions as shown by the remote expert. We propose an approach that also uses a combination of AR and VR, in a di erent manner with the VR and AR user co-located. In our approach the VR user is the one under observation. The AR user is the one observing the VR user, and can simultaneously view the physical VR user and virtual elements from the VE they are in. 3

Our proposed framework uses AR to augment the external user’s view of the VR user with elements from the VE. This enables the external AR user to view the VR user’s gestures and body language in the context of the VE. This is important as both the stimuli (virtual object) and response (VR user’s physical reaction) are involved in communicating an action-based interaction. To support this visualisation, two applications must be made, the main VR application and a companion AR application that can communicate with it.

We propose three criteria for selecting the scene elements that are visualised to the AR user: Proximity Threshold, Field of View, Importance Ranking ( gure 2). We expect di erent use cases for asymmetric interaction to require di erent selection criteria. This selection can also di er between users. We can selectively visualise objects nearby the VR user using a proximity threshold. We will implement this as the euclidean distance between the VR user’s position in space and the position of the VE scene object. This way, only the relevant object within interaction distance may be selected for the augmentation.

The Body in Cross-Reality: A Framework for Selective AR Visualisation of Virtual Objects International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal 3.2

The Field of View refers to the area of the VE that the VR user is looking at, and how close the object is to this area. This is the angle between the VR user’s forward vector and the vector between the VR user and the scene object on the yaw axis. This way, only the relevant objects in front of the VR user may be selected for augmentation. Eye tracking can potentially be used to more accurately identify which object has the VR user’s attention. 3.3

Some objects in the VE are more relevant to display to external users than others. Importance ranking sets a predetermined selection of objects that are shown to the AR user. This selection can be binary, one group of objects to visualise and another group to hide in augmentation. 3.4

The software running on the VR system (eg. HTC Vive) and AR system (eg. Microsoft Hololens) will be set up as networked applications. Two di erent versions of the application will be made, one for AR and one for VR. The VR application will be the host and contain the state of the VE. It will communicate this state as selected by the criteria described above with the AR client.

The AR application will be a client that contains minimal logic which displays the selection of objects as told by the VR host application. As users will be co-located, voice communication does not need to be networked. However, the AR user’s position must be tracked in order to support a representation of this user. Having a representation of the AR user in the VE would prevent the AR user’s voice from feeling disembodied and maximize immersion for the VR user. 4

Current VR systems involve very physical types of interaction, such as walking and picking up objects. External observers may be interested in the relationship between the body and the interactive object in a VE, and body language can provide useful information about the nature of an interaction. We present two use cases in which we propose to apply our framework and evaluate how useful gestural information is to the perception of the AR user. 4.1

The employee (VR user) is immersed in the VE to become familiarised with the workplace, and the instructor (AR user) oversees the employee’s performance in "food hygiene" from how the VR user physically handles selectively visualised food objects. • Proximity Threshold: The threshold is bound by the single station located by the VR user. Further food stations are not visualised. • Field of View: The activity of handling food only occurs within arms-reach of the VR user, thus Field of View is a less relevant criteria for this use case. • Importance Ranking: Visualising food and related tools is important to portray food handling, but other scene objects may be less relevant. 4.2

The use case simulates an architectural review in which a client (VR user) explores and critiques an interactive 1:1 scale architectural project in the VE. The architect (AR user), who is already familiar with the project, perceives only the speci c VE segments that the VR user is interacting with and referring to.

• Proximity Threshold: Less relevant as a 1:1 scale explorable architectural model is typically too large to e ectively include within a visible threshold. • Field of View: The VR user looks at di erent parts of the architectural proposal, and the AR user shares the VR user’s line of sight when referring to speci c architectural elements. • Importance Ranking: Only relevant if the architect has included interactive architectural elements, such as furniture. 5

We seek to evaluate this approach in an experimental study. We will investigate changing the selection criteria (Proximity Threshold, Field of View, Importance Ranking) across di erent use cases, as well as conducting user experience studies focusing on how the AR user perceives the activity and bodily engagement of the VR user. As a baseline, we will also compare the selective visualisation framework to two other conditions: an external user that can see the VR user without AR, and a VR spectator which can see the entire VE without seeing the VR user. 6

In this paper we presented a novel visualisation framework for selective visualisation of VE elements to an AR user. This visualisation allows the external AR spectator to view the physical movements of the VR user in context of the VE. All visual detail of the VR user’s physical appearance may be preserved, including gestures and body language, while giving the context of the actions they are performing in the VE. As such, we aim to nd the optimal amount of VE context to visualise to clearly and e ectively convey the nature of an interaction between immersed users and their environment.