Developing a Cross Reality Prototype to Enhance Workflow in Pre-Surgical Evaluations of Epilepsy Patients Brody Wells1,∗ , Nanjia Wang1 and Frank Maurer1 1 University of Calgary, 2500 University Dr NW, Calgary AB, Canada, T2N 1N4 Abstract Epilepsy is a neurological disorder that affects individuals worldwide. Accurate identification of seizure origination regions in the brain, primarily through diagnostic tools like intracranial Electroencephalo- grams (iEEG) and Magnetic Resonance Imaging (MRI) is crucial for successful surgical interventions. Current diagnostic interpretation methods pose challenges, especially in conceptualizing 4-dimensional information on a 2-dimensional screen. This research explores the potential of Cross Reality (CR) tech- nology to enhance workflow in pre-surgical evaluations of epilepsy patients. Our objectives include assessing the potential of CR technology in addressing interpretation challenges, exploring its cognitive benefits over traditional tools, understanding design considerations for clinical CR systems, and laying groundwork for future CR integration in medical diagnostics. We aim to create a visualization system that integrates a traditional computer monitor-based system that displays iEEG and MRI with Mixed Reality (MR) features, allowing pre-surgical evaluations workflow to be enhanced with the advantage brought by immersive technologies while allowing doctors to preserve the benefits of traditional visualization tools they are familiar with. This research intends to bridge gaps in literature, providing insights into the integration of CR in neurology workflows. Keywords Serious Extended Reality, Cross Reality, Mixed Reality, Visualization, Visualization Techniques, 1. Introduction Epilepsy accounts for 0.5% of the world’s disease burden and affects individuals irrespective of gender or ethnicity [1]. Surgical intervention remains a critical option when other treatments fail. Success of surgery hinges on the neurologist’s proficiency in accurately identifying the brain regions where seizures originate [2]. This identification process is primarily based on diagnostic data, such as intracranial Electroencephalograms (iEEG) and Magnetic Resonance Imaging (MRI). However, the current tools and methods employed to interpret this data present challenges. Research Objectives: The goal of this research is to explore and evaluate the potential of Cross Reality (CR) technology in enhancing clinical workflow in pre-surgical evaluations of INI-DH 2024: Workshop on Innovative Interfaces in Digital Healthcare, in conjunction with International Conference on Advanced Visual Interfaces 2024 (AVI 2024), June 3–7, 2024, Arenzano, Genoa, Italy (2024) Envelope-Open brody.wells@ucalgary.ca (B. Wells); nanjia.wang1@ucalgary.ca (N. Wang); frank.maurer@ucalgary.ca (F. Maurer) GLOBE http://www.brodyai.com/ (B. Wells); https://seriousxr.ca/ (F. Maurer) Orcid 0009-0004-4597-4383 (B. Wells) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Figure 1: Concept visualization created using OpenAI’s DALL·E model. The image was generated using the following prompt: A neurologist working in a Mixed Reality environment with MRI information on one desktop monitor and EEG data on another desktop monitor and they are interacting with an augmented model of a brain epilepsy patients. Specifically, we aim to: 1. Gain feedback on the efficacy of CR technology in addressing the challenges posed by traditional diagnostic interpretation methods. 2. Determine potential cognitive benefits of using CR over a traditional system utilizing 2D screens with mouse and keyboard, particularly in reducing cognitive load and improving data conceptualization. 3. Investigate design and user experience factors associated with CR systems in a clinical setting. 4. Establish a foundation for future research on the integration of CR technology in medical diagnostics and treatment planning. 2. Background 2.1. Milgram’s Reality-Virtuality Continuum Milgram et al. put forth the concept of the Reality-Virtuality Continuum (RVC), suggesting that the real environment, Augmented Reality (AR), Augmented Virtuality (AV), and Virtual Reality (VR) are not discrete concepts, but rather exist on a continuum of mixed realities [3]. The ideas in this paper focus on CR, which are technologies that allow users to transition between different points on the RVC or interact with multiple systems along the RVC concurrently [4]. 2.2. Medical Imaging Neurologists analyze various forms of medical data when performing pre-surgical evaluations on epileptic patients, of which MRIs and EEGs are among the most useful [5]. MRIs are used to generate 3D images of anatomical structures in the human body and are well suited for capturing soft tissue structures such as the brain [6]. Regarding the use of EEG data, this study will concentrate on stereo Electroencephalography (sEEG), a specific type of iEEG that employs minimally invasive depth electrodes implanted intracranial to record brain electrical activity. Neurologists must match the temporal data of the sEEG with the spatial MRI data to pinpoint the origins of seizure spread inside the patient’s brain [5]. At present, neurologists rely on traditional desktop monitors to analyze diagnostic medical data [7]. The challenge is the need to conceptualize 4-dimensions of information (3 spatial dimensions of the MRI plus the temporal sEEG data), on a 2-dimensional (2D) screen. This traditional setup demands significant cognitive effort [7]. 3. Related Work 3.1. Benefits of Virtual Reality Systems Aminolroaya, using their RealityFlow prototype, showed that a benefit of VR is its ability to reduce the cognitive burden for neurologists when trying to conceptualize higher-dimensional information from 2D MRI and iEEG data. The study also found that participants got a better overall picture of where the electrodes were placed and where the seizures were starting and spreading [7]. Neurologists utilize Visual Spatial Abilities (VSA), among other cognitive functions, to conceptualize patient anatomy from the 2D medical images. A scoping review found VSA has a correlation with the surgical success in simulated environments, although the results were mostly studied in novice learners and medical trainees [8]. RealityFlow also used interactive visualizations, where the user could rotate, transform, and slice through the 3D medical models [7]. Riegler et al. refer to interactive visualizations as an “assistance systems” helping to decrease cognitive load and enhance cognition [9]. The VR environment of RealityFlow also benefited from a scalable workspace. The large space allowed small multiples to be integrated into the system, aiding in the visualization of seizure data. Small multiples map time to space [10], in contrast to animations that map time to time [11], and therefore are best used when limited space is not a concern. Other VR systems have found success in presurgical planning of epilepsy patients and are evidence of the benefits. Phan et al. demonstrated successful use of VR visualization with their Surgical Theater system. In one patient case, the VR models generated from patient data were used by neurologists to both better interpret sEEG findings and also educate patient and family members about proposed surgery. The VR models were also used by the neurosurgeon to delineate areas of focus [12]. 3.2. Limitations of Virtual Reality Systems VR is not without its limitations, with issues such as reduced readability, limited precision in movements, and worse depth perception than AR [13]. These limitations often make certain tasks more effectively handled on conventional desktop monitors. Aminolroaya’s study found evidence it would be difficult to switch between using their conventional desktop monitor and working with RealityFlows VR environment [7]. Some tasks, especially those requiring high accuracy, are still best done using tools such as a mouse and keyboard [14]. Wang et al. found similar feedback regarding VR limitations during their pilot study using CR in cardiac surgery planning. Participants felt that an immersive VR system would never replace the traditional clinical workflow using a conventional desktop monitor [13]. 3.3. Cross Reality Potentials Benko et al. demonstrated the potential for a CR system to harness benefits from multiple points along the RVC, by using a ”pull” gesture to transition objects on a 2D touchscreen into a 3D MR environment [15]. Riegler et al. showed how different points on the RVC can offer different advantages such as the high readability of 2D screens or the stereoscopic presentation of AR and VR [9]. One challenge for CR system design is creating transitions along the RVC that don’t negatively impact usability. Feld et al. found that fast and efficient transitions were preferred when cognitive tasks needed to be performed [16]. The ability to transition from the real environment to AR or VR gives CR systems the benefit of scalable workspaces, which introduces specific design space considerations. Reipschlager et al. discussed considerations with large interactive AR systems noting that managing data density and complexity along with perceptual issues must be carefully thought through [17]. Shupp et al. showed curved AR screens keep data along the user’s periphery, within their perception, and can aid in accomplishing certain tasks in AR [18]. Reipschlager noted that when number of visuals is small a flat layout is best no matter what the user’s preference is [17]. 4. Contrasting Our Work Our primary objective is to investigate the potential of CR systems in enhancing neurologists’ workflow. While the literature has highlighted the benefits and challenges of systems that adhere to a single point on the RVC, our research aims to bridge this gap. We’re prototyping a system that doesn’t replace traditional methods, but will attempt to integrate them with MR tools. By allowing neurologists to transition between modalities, we are addressing a crucial limitation identified in previous studies. We hope valuable feedback and observations will be gained to expand on the overall literature. 5. Requirements Elicitation An initial elicitation interview was conducted with neurologists with experience conducting pre-surgical evaluations of epilepsy patients. Open ended questions were asked of the experts to gather information on key points. From this we developed an initial set of requirements for the prototype. 1. An augmented brain model constructed from a patient’s MRI data, that can be interacted with in MR and linked to views on the traditional software. 2. A semi-transparent model can be used to view and select electrodes, which will either update slicing location of their 2D MRI software or the slicing location of a second, higher-definition augmented brain model. 3. Intuitive interaction methods, ideally using hand gestures to move, rotate, resize the augmented model and also to select individual electrodes. 6. Prototype Design We developed an early-stage prototype that can be shown to neurologists to gain further feed- back. Allowing neurologists to test the prototype based on their initial elicitation requirements will enable further, more refined feedback and could prompt changes in how they envision CR technologies being integrated into their workflow. 6.1. Technologies Used The application for the prototype was created using Unity (2022.3 LTS)1 . Two separate unity applications were created, one which controlled all features relating to the MR environment, and the other application simulated desktop MRI software. These two applications were linked over a network using Unity Netcode2 , allowing for CR interactions where changes made to the MR models would affect the views displayed on the desktop software. (See Figure 2.) To allow the users to both interact with augmented models in MR space and also view and use their desktop software, we chose to use the Varjo XR33 HMD. The Varjo XR3 HMD provides 12-megapixel video pass-through, 115 degrees field of view, and 70 pixels per degree (The highest ppd of devices currently on the market). We added hand tracking and gesture recognition using UltraLeaps4 hand tracking package for Unity. This allowed the prototype to be used without the aid of any controllers. The 3D brain models were generated using VolumeViewerPro 5 , a Unity asset that can convert NIfTI formatted MRI files to 3D objects using volumetric ray tracing. 6.2. Simulated Desktop Software An application was created to simulate software for viewing MRI and EEG data. The application consisted of three orthogonal views (axial, coronal, and sagittal) positioned along the top of the user’s screen. The three views would be replaced by one central view if the user was using the ”electrode selector” feature (refer to features section). Along the bottom of the screen were two mock electrode strips which would be highlighted green to indicate which of the two electrodes in our model was selected. 1 https://unity.com/ 2 https://unity.com/products/netcode 3 https://varjo.com/products/varjo-xr-3/ 4 https://www.ultraleap.com/tracking/ 5 https://liscintec.com/shaders/ Figure 2: A series of images depicting a CR interaction. (a) The user employs a ”pinch” gesture near the blue slicing plane to slide it forward in MR space along the selected axis, positioning the coronal plane anteriorly towards the frontal cortex. (b) The plane has been moved posterior using the pinch gesture. A yellow arrow highlights the simultaneous update of the monitor display with the MR interactions. The monitor display in the background is simulating traditional software depicting the three orthogonal views of the MRI data. 6.3. Features A menu was anchored to the user’s palm to control the CR features (See Figure 3). Users can adjust the positions of the axial, coronal, and sagittal views, or anchor the planar view to a selected electrode, allowing them to rotate the plane relative to that electrode (electrode positions were simulated via pre-placed unity objects in the 3D brain model). Changes made in MR would be reflected in the views shown on the conventional desktop monitor. There is also ”culling cube” users could move freely to slice into the MR brain from any angle. The users could change the size, transparency, contrast, and brightness of the 3D brain model to suit their task needs. 6.4. Hand Interactions All interactions in the CR prototype were accomplished without controllers, by using hand interactions. There were three gestures the users could perform: a grab, pinch, or a point (See Figure 4). Grabbing was used to interact with the 3D brain model and culling cube. When the user grabs one of these objects they can freely move it around in MR as the object tracks the position and rotation of their fist. Pinching was used to interact with any of the slicing planes. If the user performed a pinch near a plane, the plane would continue to track its location to the user’s index finger until they release the pinch. Pointing was used for interacting with the control menu buttons and sliders, as well as for selecting electrodes in the 3D brain model. The user could select electrodes in the brain model by pointing at the desired electrode. On the control menu a user could press a button with their fingertip, or move a slider by depressing it with their finger then sliding it left or right. Figure 3: Control panel anchored to the user’s palm, displayed when the palm faces the user. The buttons are organized as follows: Starting from the top left to the bottom right, there are ”Cull” buttons to activate culling mode for each orthogonal plane; ”Axial,” ”Coronal,” and ”Sagittal” buttons to toggle the visibility of each plane; ”Toggle brain” to show or hide the 3D brain model; ”Electrode selector” to initiate electrode selection mode; ”Adjust visuals” to reveal the visual adjustment sliders; and ”Cube” to manage slicing cube visibility. Slider bars, arranged from top to bottom, control size, transparency, contrast, and brightness. Figure 4: The three hand gestures used for MR interactions: (a) Grab, (b) Point, and (c) Pinch. 7. Discussion and Future Work The suite of software currently used by neurologists has evolved over decades, with specialists becoming proficient with traditional interfaces. CR may enable neurologists to use innovative interaction and data analysis methods offered by MR, while retaining familiar software tools. This hybrid model has potential to enhance the neurologists’ workflow without disrupting their established practices. A complete CR system requires direct integration with existing neurology software, which presents many hurdles. Additionally, precise localization of electrodes and medically accurate visualization of anatomy will pose their own challenges. Our prototype manually placed 3D objects on a brain model to represent these locations, but an ideal system would automatically detect these positions from MRI data. To evaluate our prototype CR system we will conduct demonstration sessions with neurol- ogists. These sessions will involve a detailed demonstration of the prototype’s functionality, followed by a hands-on testing period where neurologists can interact with and explore the tool’s features. Feedback will be collected through open-ended discussions to guide the refine- ment of our prototype, ensuring it aligns closely with the neurologists’ needs and workflow requirements. The insights gained will direct future enhancements and iterations of the system. References [1] E. Beghi, The epidemiology of epilepsy, Neuroepidemiology 54 (2020) 185–191. URL: https://www.karger.com/Article/FullText/503831. doi:10.1159/000503831 . [2] J. F. Téllez-Zenteno, R. Dhar, S. Wiebe, Long-term seizure outcomes fol- lowing epilepsy surgery: a systematic review and meta-analysis, Brain 128 (2005) 1188–1198. URL: http://academic.oup.com/brain/article/128/5/1188/278443/ Longterm-seizure-outcomes-following-epilepsy. doi:10.1093/brain/awh449 . [3] P. Milgram, H. Takemura, A. Utsumi, F. Kishino, Augmented reality: a class of displays on the reality-virtuality continuum, in: H. Das (Ed.), Telemanipulator and Telepresence Technologies, volume 2351, International Society for Optics and Photonics, SPIE, 1995, pp. 282 – 292. URL: https://doi.org/10.1117/12.197321. doi:10.1117/12.197321 . [4] N. Wang, F. Maurer, A design space for single-user cross-reality applications, in: Proceedings of the 2022 International Conference on Advanced Visual Interfaces, AVI 2022, Association for Computing Machinery, New York, NY, USA, 2022. URL: https: //doi.org/10.1145/3531073.3531116. doi:10.1145/3531073.3531116 . [5] W. Wilson, Intracranial EEG-fMRI analysis of interictal markers in drug-resistant focal epilepsy, Doctoral dissertation, University of Calgary, 2023. URL: https://prism.ucalgary. ca/handle/1880/116802, available online at https://dx.doi.org/10.11575/PRISM/41644. [6] N. I. of Biomedical Imaging, Bioengineering, Magnetic resonance imag- ing(mri), n.d. URL: https://www.nibib.nih.gov/science-education/science-topics/ magnetic-resonance-imaging-mri, accessed: 24 October 2023. [7] Z. Aminolroaya, Streamlining the Epilepsy Pre-surgical Evaluation Workflow with Virtual Reality, Ph.D. thesis, University of Calgary, Calgary, 2023. URL: https://dx.doi.org/10.11575/ PRISM/dspace/41012. [8] P. Kalun, K. Dunn, N. Wagner, T. Pulakunta, R. Sonnadara, Recent evidence on visual-spatial ability in surgical education: A scoping review: Des preuves récentes sur les habiletés visuo- spatiales pour la formation en chirurgie : revue exploratoire, Canadian Medical Education Journal (2020). URL: https://journalhosting.ucalgary.ca/index.php/cmej/article/view/69051. doi:10.36834/cmej.69051 . [9] A. Riegler, C. Anthes, H.-C. Jetter, C. Heinzl, C. Holzmann, J. Herbert, M. Brunner, S. Auer, J. Friedl, B. Fröhler, C. Leitner, F. Pointecker, D. Schwajda, S. Tripathi, Cross-virtuality visualization, interaction and collaboration, in: Proceedings of the 2020 ACM Interactive Surfaces and Spaces (ACM ISS 2020), 2020. [10] E. R. Tufte, The visual display of quantitative information, The Journal for Healthcare Quality (JHQ) 7 (1985) 15. [11] W. Aigner, S. Miksch, H. Schumann, C. Tominski, Visualization of Time-oriented Data, volume 4, Springer, 2011. doi:https://doi.org/10.1007/978- 0- 85729- 079- 3 . [12] T. N. Phan, K. J. Prakash, R.-J. S. Elliott, A. Pasupuleti, W. D. Gaillard, R. F. Keating, C. O. Oluigbo, Virtual reality–based 3-dimensional localization of stereotactic eeg (seeg) depth electrodes and related brain anatomy in pediatric epilepsy surgery, Child’s nervous system 38 (2022) 537–546. [13] N. Wang, F. Maurer, User-Centered Prototyping for Single-User Cross-Reality Virtual Object Transitions, in: 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), IEEE, Singapore, Singapore, 2022, pp. 171–174. URL: https://ieeexplore.ieee.org/document/9974379/. doi:10.1109/ISMAR- Adjunct57072.2022. 00039 . [14] X. Wang, L. Besançon, M. Ammi, T. Isenberg, Understanding differences between com- binations of 2d and 3d input and output devices for 3d data visualization, International Journal of Human-Computer Studies 163 (2022) 102820. URL: https://linkinghub.elsevier. com/retrieve/pii/S1071581922000490. doi:10.1016/j.ijhcs.2022.102820 . [15] H. Benko, E. Ishak, S. Feiner, Cross-dimensional gestural interaction techniques for hybrid immersive environments, in: IEEE Proceedings. VR 2005. Virtual Reality, 2005., IEEE, Bonn, Germany, 2005, pp. 209–327. URL: http://ieeexplore.ieee.org/document/1492776/. doi:10.1109/VR.2005.1492776 . [16] N. Feld, P. Bimberg, B. Weyers, D. Zielasko, Keep it simple? evaluation of transitions in virtual reality, in: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA ’23, Association for Computing Machinery, New York, NY, USA, 2023, pp. 1–7. URL: https://dl.acm.org/doi/10.1145/3544549.3585811. doi:10.1145/ 3544549.3585811 . [17] P. Reipschlager, T. Flemisch, R. Dachselt, Personal augmented reality for information visu- alization on large interactive displays, IEEE Transactions on Visualization and Computer Graphics 27 (2021) 1182–1192. doi:10.1109/TVCG.2020.3030460 . [18] L. Shupp, C. Andrews, M. Dickey-Kurdziolek, B. Yost, C. North, Shaping the display of the future: The effects of display size and curvature on user performance and insights, Human–Computer Interaction 24 (2009) 230–272. doi:10.1080/07370020902739429 .