The Social and hUman ceNtered XR (SUN) project is focused on developing eXtended Reality (XR) solutions that integrate the physical and virtual world in a way that is convincing from a human and social perspective. In this paper, we outline the limitations that the SUN project aims to overcome, including the lack of scalable and cost-efective solutions for developing XR applications, limited solutions for mixing the virtual and physical environment, and barriers related to resource limitations of end-user devices. We also propose solutions to these limitations, including using artificial intelligence, computer vision, and sensor analysis to incrementally learn the visual and physical properties of real objects and generate convincing digital twins in the virtual environment. Additionally, the SUN project aims to provide wearable sensors and haptic interfaces to enhance natural interaction with the virtual environment and advanced solutions for user interaction. Finally, we describe three real-life scenarios in which we aim to demonstrate the proposed solutions.
1 and
Giuseppe Amato1
investigating and developing extended reality (XR) solutions that integrate the physical and the virtual world in a convincing way, from a human and social perspective.
More in detail, the relevant limitations that we will address are: • Lack of solutions to develop scalable and costefective new XR applications : building an XR application for a new physical environment requires creating from scratch its accurate digital twin, with both visual and physical/functional properties, with significant eforts and high costs. • Lack of convincing solutions for mixing the virtual and physical environment : generally, augmented reality is limited to the visual alignment of physical and virtual components. However, a convincing XR requires physical and virtual elements nized by CINI, May 29–31, 2023, Pisa, Italy ∗Corresponding author. vironments – The system will learn from the physical world, during its use. Like a baby, who learns how to recognize and use objects with the experience, SUN will use artificial intelligence, computer vision, and sensor analysis to incrementally learn, during its usage, the visual and physical properties of real objects, and to generate, recognize, and use digital twins in the virtual environment. – Learned objects and environments will be incrementally added to the SUN Digital
• Seamless and convincing interaction between the and thermal cues under fingertips) for XR physical and the virtual world scenarios, such as interaction with 3D vir– Objects and environments of the physical tual objects.
world have a digital twin in the virtual – We will provide advanced solutions for world, with the same physical and visual user interaction, including gaze-based and properties. gesture-based interaction. – Manipulating an object in the physical • Artificial intelligence-based solutions to address world will have the same plausible efect in current computing, memory, and network limitathe virtual world. For instance, a physical tions of wearable devices wrench could be used to unscrew a virtual bolt. Complementarily, a virtual wrench – AI will be used to generate plausible, highmanipulated in the virtual world, will pro- quality renderings also for coarse-grained, vide the user with a feeling consistent with low-resolution, or incomplete 3D models, a physical wrench. leveraging on solutions similar to those used for deep-fake generation. • Wearable sensors and haptic interfaces for convincing and natural interaction with the virtual The solutions will be demonstrated in three real-life environment scenarios. The rest of the paper is organized as follows: – Wearable haptics will enhance physical in- Section 2 describes the scenarios. Section 3 presents the teraction with virtual menus and contex- technical parts of the project. Finally, Section 4 concludes tual information displayed to the user. In the paper. addition, solutions for body contextual information (such as skin stretch-vibration on body parts) will lightly guide the user in 2. Scenarios tasks such as remote training, home physical exercises, and interaction with other The project will assess the developed technologies in persons. these three real-life scenarios: – Novel wearable haptic interfaces will al- 1. Extended reality for rehabilitation.
low multisensory feedback (vibrotactile 2. Extended reality for safety and social interaction improvement, enhancing accuracy as well as portability at work. in the home environment. Starting from a visual rep3. Extended reality for people with serious mobility resentation of an exercise, delivered through an avatar and verbal communication diseases. of the therapist performing it in the correct way, the patient will be called to repeat it, or in sync with the
XR experiments are going to be good examples to help avatar performing the exercise. Wearables and wireless
better understand the social impact of those technologies sensors will be used to measure and deliver contextual
even considering that COVID-19 boosted a positive atti- information, including multiple inertial measurement
tude toward them [
Rehabilitation adherence and fidelity are especially challenging, alongside motor learning with personalized feed- 2.2. Extended reality for safety and social back. Physiotherapy after orthopedic surgery and other interaction at work kinds of upper limb motor impairment such as “frozen shoulder” or severe hand arthritis is crucial for complete rehabilitation but is often repetitive, tedious, and timeconsuming. Actually, in order to achieve motor recovery a very long physiotherapy treatment, sometimes more than 50 sessions, is needed. Therefore, there is a necessity to suitably address the evidence-practice gap and translate digital innovations into practice while enabling their AR and VR can create more immersive experiences for people at work in order to make their job safer, by providing new ways to be aware of possible hazards and receive more efective, engaging and entertaining training on safety procedures. This is to alert and prevent serious accidents provoked by the co-occurrences of diferent causes, which can be avoided by conscious collaboration.
With VR/AR headsets, workers will be able, for example, to better understand dificult-to-grasp concepts or 2.3. Extended reality for people with topics such as protocols and procedures for safety and serious mobility and verbal security. It ofers a great opportunity for the industry communication diseases to be able to make use of a variety of diferent options relying on extended reality experiences to provide each Some people with various motor disabilities or after worker with an immersive access to relevant informa- strokes have huge dificulties in communicating with tion, and to encourage the adoption of safer behaviours. each other and even to address their vital needs. The Extended reality encompassing virtual, augmented, and project will join the challenge to find a dedicated commixed reality brings immersive experiences to workers munication pathway for those people introducing the no matter where they are. A XR experience can “come possibility to interact with some specific social cues and alive” when a worker puts on a VR headset and walks, for transform them in clear communication or actions. The instance, on a shop floor. Workers can experience hard- proposal is to count on residual abilities giving them a to-conceptualise current-day topics through extended meaning in terms of communication supported when reality, such as moving around in potentially dangerous needed by avatars in a virtual environment. The objecenvironments. XR technology will improve efectiveness tive is to realise low-cost non-invasive tools based for and user engagement. Through an adequate design of instance on the existing biofeedback, face expression XR contents, immersive experience seamlessly integrates and other input. The challenge will be to create a soseveral principles not present when using non-immersive lution allowing the design of person specific extended contents such as better contextualization and real-time reality interaction even at home in the working place or decision-making feedback, in a safe yet realistic environ- at school and developing novel multi-user virtual comment for practice. The Holo-Light’s software Engineering munication and collaboration solutions that provide coSpace AR 3S, which allows workers to visualise and work herent multisensory experiences and optimally convey with design files including CAD in an XR environment, relevant social cues. Successful implementation of the will be optimised and upgraded to encourage social inter- pilot for this scenario allows people with communicaaction among workers to also provide them with alerts tion and motor disabilities to interact with friends and and indications about the possibility of incidents. Cor- relatives, meeting realistically in an extended (physical + rect and incorrect behaviour will be simulated. Overall, virtual) environment. The person with communication it will enable designing, prototyping, quality assurance, and motor disabilities will be represented by an avatar and overall technical industrial training and education which will interact with other people staying in a physiin emerging manufacturing technology scenarios also cal augmented environment. Simultaneously, the person with the opportunity to not just increase the speed in with communication and motor disabilities will have the acquiring new tasks or to improve industrial processes illusion to be also in the same physical environment with but overall, also increasing the safety for workers dealing the other people, and can interact with the new interwith complex machines. Features that will be exploited faces ofered by SUN. SUN, in fact, will develop a new in the scenario include: generation of non-invasive bidirectional body-machine interfaces (BBMIs), which will allow a smooth and very • High Quality 3D Content in XR: To securely im- efective interaction of people with diferent types of port and view all your design files for AM in XR. sensory-motor disabilities with virtual reality and avatars. Visualize and manipulate even data-intensive 3D The BBMI will be able to decode motor information by content. using arrays of printed electrodes to record muscular • Merge the Real and Virtual: Place your CAD/engi- activities and inertial sensors. The position of the senneering designs in the real world. Manipulate sors will be customised according to the specific motor projected assemblies directly at their envisaged abilities of the subjects. For example, it could be possible destination. to use shoulder and elbow movements, and it could be • Work on Complex Holograms: Fully manipulate also possible to record muscular activities from auricular your designs and assemblies. Place, rotate, adjust, muscles. All these sensors will be used to record inforresize, slice and dice. Save your work. mation during diferent upper limb and hand functions • Share the Experience: Collaborate with your col- movements. We will use a procedure to identify the sigleagues, partners, or customers in AR. Set up local nals more useful for the diferent tasks and implement and global meetings in an XR environment. a dedicated decoding algorithm. This approach allowed developing a simplified and yet very efective approach to control flying drones and will probably provide very interesting results also in this case. A machine learning approach will be implemented for the decoding of the diferent tasks relying on human-machine interfaces and in general on decoding information from electrophysiological and biomechanical signals. SUN will also allow to give sensory feedback to the user about the movement and the interaction with the avatar in the virtual environment, using transcutaneous electrical stimulation or small actuators for vibration.
Various techniques exist that allow recognizing
physical objects and environments and linking/registering
them with the corresponding virtual ones to enable
taskspecific interactions between users and augmented
environments. Existing techniques are based on image
classification, object detection, and semantic
segmentation via deep neural networks [
In the context of the project, we will advance the current state of the art by developing AI-based solutions that allow the incremental and automatic discovery of environments, objects, and object usage patterns, while the system is being used. This will permit the incremental creation of libraries of reusable visually recognizable virtual objects and environments, the cost reduction in developing extended reality applications, and the scalability of extended reality to large scenarios.
Currently, most of the eforts to create digital twins
representation focus on the recreation of the shape and
appearance of an existing object. However, the mechanical
characteristics of a digital representation drive their
actual physical behavior and are necessary for interactive
environments to maintain the perceived plausibility of
the representation. Obtaining such information
practically and intuitively is a technical challenge [
The main objective is to provide innovative techniques to acquire and estimate the mechanical properties (like mass distribution or stifness/softness) of 3D objects. This will be done by integrating wearable fingertip-sensing devices with maker-based video tracking during controlled object manipulation. AI data interpretation, followed by multimodal data fusion, will allow learning of the expected inertia tensor and other physical properties that best match the expected sensor readings. An important aspect is the design and 3D printing of objects with controlled mechanical properties for the acquisition of datasets with ground truth data to use during the learning phase.
XR environments are populated by 3D models representing real-world objects (assets). Assets coming from stock sources (commercial or open repositories) are often used to cope with the general situation, but for specific scenarios, it is often necessary to produce on the fly new 3D models. This task will address this phase of asset creation.
A first direction to pursue is to help in producing more
complete and convincing 3D models by adding AI
processing to the diferent steps of the acquisition pipeline.
To this aim, the photogrammetric reconstruction seems
the most promising one: AI processing can be efectively
used in removing from photos those problems that
reduce the accuracy of alignment and 3D data generation
or unwanted areas [
On a diferent level, for specific classes of objects, we can exploit the a priori knowledge of the domain, to pre-train networks on examples. This will ensure better support in the 3D model generation and a more coherent and realistic completion of the models’ geometries and textures.
that has been revamped in the deep learning era, and
automatic model instantiation has similarly received
academic and industrial attention [
We will use reinforcement learning for automatic exploration (e.g., by unmanned vehicles) of unknown and possibly large objects and environments, while minimizing the scenario discovery time. To keep acquisition feasible in a time-constrained context and allow an immersive experience from the very start of the scenario exploration, our solution will also deploy pre-built 3D model instances for objects whose complete reconstruction would need to acquire information from diferent points of view, or explore novel methods (e.g., conditional denoising difusion probabilistic models) to autogenerate the underlying representation.
The Social and Human-centered XR project aims to overcome the current limitations in developing XR solutions that integrate the physical and virtual world from a human and social perspective. The limitations addressed in the SUN project include lack of solutions to develop scalable and cost-efective new XR applications, lack of convincing solutions for mixing the virtual and physical environment, lack of plausible and convincing human interaction interfaces in XR, and barriers due to resource limitations of end-user devices. The SUN project proposes scalable solutions to obtain plausible and convincing virtual copies of physical objects and environments, seamless and convincing interaction between the physical and virtual world, wearable sensors and haptic interfaces for convincing and natural interaction with the virtual environment, and artificial intelligence-based solutions to address current computing, memory, and network limitations of wearable devices.
zon Europe Research & Innovation Programme under Grant agreement N. 101092612 (Social and hUman ceNtered XR - SUN project) and by PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - ”FAIR - Future Artificial Intelligence Research” - Spoke 1 ”Humancentered AI”, funded by the European Union under the NextGeneration EU programme.