Epilepsy is a neurological disorder that afects individuals worldwide. Accurate identification of seizure origination regions in the brain, primarily through diagnostic tools like intracranial Electroencephalograms (iEEG) and Magnetic Resonance Imaging (MRI) is crucial for successful surgical interventions. Current diagnostic interpretation methods pose challenges, especially in conceptualizing 4-dimensional information on a 2-dimensional screen. This research explores the potential of Cross Reality (CR) technology to enhance workflow in pre-surgical evaluations of epilepsy patients. Our objectives include assessing the potential of CR technology in addressing interpretation challenges, exploring its cognitive benefits over traditional tools, understanding design considerations for clinical CR systems, and laying groundwork for future CR integration in medical diagnostics. We aim to create a visualization system that integrates a traditional computer monitor-based system that displays iEEG and MRI with Mixed Reality (MR) features, allowing pre-surgical evaluations workflow to be enhanced with the advantage brought by immersive technologies while allowing doctors to preserve the benefits of traditional visualization tools they are familiar with. This research intends to bridge gaps in literature, providing insights into the integration of CR in neurology workflows.
CEUR ceur-ws.org
challenges.
LGOBE
(F. Maurer)
Epilepsy accounts for 0.5% of the world’s disease burden and afects individuals irrespective of
gender or ethnicity [
Milgram et al. put forth the concept of the Reality-Virtuality Continuum (RVC), suggesting that
the real environment, Augmented Reality (AR), Augmented Virtuality (AV), and Virtual Reality
(VR) are not discrete concepts, but rather exist on a continuum of mixed realities [
Neurologists analyze various forms of medical data when performing pre-surgical evaluations
on epileptic patients, of which MRIs and EEGs are among the most useful [
Aminolroaya, using their RealityFlow prototype, showed that a benefit of VR is its ability to
reduce the cognitive burden for neurologists when trying to conceptualize higher-dimensional
information from 2D MRI and iEEG data. The study also found that participants got a better
overall picture of where the electrodes were placed and where the seizures were starting
and spreading [
RealityFlow also used interactive visualizations, where the user could rotate, transform, and
slice through the 3D medical models [
The VR environment of RealityFlow also benefited from a scalable workspace. The large
space allowed small multiples to be integrated into the system, aiding in the visualization of
seizure data. Small multiples map time to space [
Other VR systems have found success in presurgical planning of epilepsy patients and are evidence of the benefits. Phan et al. demonstrated successful use of VR visualization with their Surgical Theater system. In one patient case, the VR models generated from patient data were used by neurologists to both better interpret sEEG findings and also educate patient and family members about proposed surgery. The VR models were also used by the neurosurgeon to delineate areas of focus [12].
VR is not without its limitations, with issues such as reduced readability, limited precision in movements, and worse depth perception than AR [13]. These limitations often make certain tasks more efectively handled on conventional desktop monitors.
Aminolroaya’s study found evidence it would be dificult to switch between using their
conventional desktop monitor and working with RealityFlows VR environment [
Wang et al. found similar feedback regarding VR limitations during their pilot study using CR in cardiac surgery planning. Participants felt that an immersive VR system would never replace the traditional clinical workflow using a conventional desktop monitor [ 13].
Benko et al. demonstrated the potential for a CR system to harness benefits from multiple
points along the RVC, by using a ”pull” gesture to transition objects on a 2D touchscreen into
a 3D MR environment [15]. Riegler et al. showed how diferent points on the RVC can ofer
diferent advantages such as the high readability of 2D screens or the stereoscopic presentation
of AR and VR [
The ability to transition from the real environment to AR or VR gives CR systems the benefit of scalable workspaces, which introduces specific design space considerations. Reipschlager et al. discussed considerations with large interactive AR systems noting that managing data density and complexity along with perceptual issues must be carefully thought through [17]. Shupp et al. showed curved AR screens keep data along the user’s periphery, within their perception, and can aid in accomplishing certain tasks in AR [18]. Reipschlager noted that when number of visuals is small a flat layout is best no matter what the user’s preference is [ 17].
Our primary objective is to investigate the potential of CR systems in enhancing neurologists’ workflow. While the literature has highlighted the benefits and challenges of systems that adhere to a single point on the RVC, our research aims to bridge this gap. We’re prototyping a system that doesn’t replace traditional methods, but will attempt to integrate them with MR tools. By allowing neurologists to transition between modalities, we are addressing a crucial limitation identified in previous studies. We hope valuable feedback and observations will be gained to expand on the overall literature.
An initial elicitation interview was conducted with neurologists with experience conducting pre-surgical evaluations of epilepsy patients. Open ended questions were asked of the experts to gather information on key points. From this we developed an initial set of requirements for the prototype.
1. An augmented brain model constructed from a patient’s MRI data, that can be interacted with in MR and linked to views on the traditional software. 2. A semi-transparent model can be used to view and select electrodes, which will either update slicing location of their 2D MRI software or the slicing location of a second, higher-definition augmented brain model. 3. Intuitive interaction methods, ideally using hand gestures to move, rotate, resize the augmented model and also to select individual electrodes.
We developed an early-stage prototype that can be shown to neurologists to gain further feedback. Allowing neurologists to test the prototype based on their initial elicitation requirements will enable further, more refined feedback and could prompt changes in how they envision CR technologies being integrated into their workflow.
The application for the prototype was created using Unity (2022.3 LTS)1. Two separate unity applications were created, one which controlled all features relating to the MR environment, and the other application simulated desktop MRI software. These two applications were linked over a network using Unity Netcode2, allowing for CR interactions where changes made to the MR models would afect the views displayed on the desktop software. (See Figure 2.)
To allow the users to both interact with augmented models in MR space and also view and use their desktop software, we chose to use the Varjo XR3 3 HMD. The Varjo XR3 HMD provides 12-megapixel video pass-through, 115 degrees field of view, and 70 pixels per degree (The highest ppd of devices currently on the market).
We added hand tracking and gesture recognition using UltraLeaps4 hand tracking package for Unity. This allowed the prototype to be used without the aid of any controllers.
The 3D brain models were generated using VolumeViewerPro 5, a Unity asset that can convert NIfTI formatted MRI files to 3D objects using volumetric ray tracing.
An application was created to simulate software for viewing MRI and EEG data. The application consisted of three orthogonal views (axial, coronal, and sagittal) positioned along the top of the user’s screen. The three views would be replaced by one central view if the user was using the ”electrode selector” feature (refer to features section). Along the bottom of the screen were two mock electrode strips which would be highlighted green to indicate which of the two electrodes in our model was selected. 1https://unity.com/ 2https://unity.com/products/netcode 3https://varjo.com/products/varjo-xr-3/ 4https://www.ultraleap.com/tracking/ 5https://liscintec.com/shaders/
A menu was anchored to the user’s palm to control the CR features (See Figure 3). Users can adjust the positions of the axial, coronal, and sagittal views, or anchor the planar view to a selected electrode, allowing them to rotate the plane relative to that electrode (electrode positions were simulated via pre-placed unity objects in the 3D brain model). Changes made in MR would be reflected in the views shown on the conventional desktop monitor. There is also ”culling cube” users could move freely to slice into the MR brain from any angle. The users could change the size, transparency, contrast, and brightness of the 3D brain model to suit their task needs.
All interactions in the CR prototype were accomplished without controllers, by using hand interactions. There were three gestures the users could perform: a grab, pinch, or a point (See Figure 4).
Grabbing was used to interact with the 3D brain model and culling cube. When the user grabs one of these objects they can freely move it around in MR as the object tracks the position and rotation of their fist.
Pinching was used to interact with any of the slicing planes. If the user performed a pinch near a plane, the plane would continue to track its location to the user’s index finger until they release the pinch.
Pointing was used for interacting with the control menu buttons and sliders, as well as for selecting electrodes in the 3D brain model. The user could select electrodes in the brain model by pointing at the desired electrode. On the control menu a user could press a button with their ifngertip, or move a slider by depressing it with their finger then sliding it left or right.
The suite of software currently used by neurologists has evolved over decades, with specialists becoming proficient with traditional interfaces. CR may enable neurologists to use innovative interaction and data analysis methods ofered by MR, while retaining familiar software tools. This hybrid model has potential to enhance the neurologists’ workflow without disrupting their established practices.
A complete CR system requires direct integration with existing neurology software, which presents many hurdles. Additionally, precise localization of electrodes and medically accurate visualization of anatomy will pose their own challenges. Our prototype manually placed 3D objects on a brain model to represent these locations, but an ideal system would automatically detect these positions from MRI data.
To evaluate our prototype CR system we will conduct demonstration sessions with neurologists. These sessions will involve a detailed demonstration of the prototype’s functionality, followed by a hands-on testing period where neurologists can interact with and explore the tool’s features. Feedback will be collected through open-ended discussions to guide the refinement of our prototype, ensuring it aligns closely with the neurologists’ needs and workflow requirements. The insights gained will direct future enhancements and iterations of the system. [12] T. N. Phan, K. J. Prakash, R.-J. S. Elliott, A. Pasupuleti, W. D. Gaillard, R. F. Keating, C. O.
Oluigbo, Virtual reality–based 3-dimensional localization of stereotactic eeg (seeg) depth electrodes and related brain anatomy in pediatric epilepsy surgery, Child’s nervous system 38 (2022) 537–546. [13] N. Wang, F. Maurer, User-Centered Prototyping for Single-User Cross-Reality Virtual Object Transitions, in: 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), IEEE, Singapore, Singapore, 2022, pp. 171–174. URL: https://ieeexplore.ieee.org/document/9974379/. doi:10.1109/ISMAR-Adjunct57072.2022. 00039. [14] X. Wang, L. Besançon, M. Ammi, T. Isenberg, Understanding diferences between combinations of 2d and 3d input and output devices for 3d data visualization, International Journal of Human-Computer Studies 163 (2022) 102820. URL: https://linkinghub.elsevier. com/retrieve/pii/S1071581922000490. doi:10.1016/j.ijhcs.2022.102820. [15] H. Benko, E. Ishak, S. Feiner, Cross-dimensional gestural interaction techniques for hybrid immersive environments, in: IEEE Proceedings. VR 2005. Virtual Reality, 2005., IEEE, Bonn, Germany, 2005, pp. 209–327. URL: http://ieeexplore.ieee.org/document/1492776/. doi:10.1109/VR.2005.1492776. [16] N. Feld, P. Bimberg, B. Weyers, D. Zielasko, Keep it simple? evaluation of transitions in virtual reality, in: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA ’23, Association for Computing Machinery, New York, NY, USA, 2023, pp. 1–7. URL: https://dl.acm.org/doi/10.1145/3544549.3585811. doi:10.1145/ 3544549.3585811. [17] P. Reipschlager, T. Flemisch, R. Dachselt, Personal augmented reality for information visualization on large interactive displays, IEEE Transactions on Visualization and Computer Graphics 27 (2021) 1182–1192. doi:10.1109/TVCG.2020.3030460. [18] L. Shupp, C. Andrews, M. Dickey-Kurdziolek, B. Yost, C. North, Shaping the display of the future: The efects of display size and curvature on user performance and insights, Human–Computer Interaction 24 (2009) 230–272. doi:10.1080/07370020902739429.