This research presents a groundbreaking approach to diagnosing Neurological Evaluations by incorporating Augmented Reality (AR) into a multi-modal strategy. We are especially focused on using AR goggles to capture the symptoms of neurological diseases in detail, revolutionizing our understanding of disorders afecting the brain. Our method relies on detailed analysis of sensor data and the nuances of movements such as hand actions, walking patterns, and eye movements, ofering a novel perspective on patient care. At the heart of our innovation is a self-guided, game-alike experience lasting 30 minutes, consisting of 17 diverse activities that integrate motor, walking, and eye exercises, along with speech, memory, and cognitive challenges. This interactive platform, together with traditional clinical assessments, provides a comprehensive evaluation of the patient's physical and mental abilities. By automating the tasks and using AR glasses to improve patient comfort, our method not only engages patients efectively but also stands out as a powerful tool for diagnosis, ushering in a new age of interactive neurodiagnostics that combines technological innovation with patient involvement.
Enhanced Neurological Evaluations Using Augmented Reality for neurological assessment
encapsulates the dual application of augmented reality technology in both the diagnosis and
monitoring of neurodegenerative diseases, such as Parkinson’s Disease (PD), and in the
rehabilitation of stroke survivors. Neurodegenerative disorders, characterized by their progressive
nature, afect a considerable portion of the population, with PD impacting about 0.15-0.30% of
all individuals and approximately 1.5-2% of those aged 70 or older [
Addressing these challenges, this study proposes a novel application of Augmented Reality (AR) through a multi-modal approach, aiming not only to refine the diagnostic process for neurodegenerative diseases but also to revolutionize the way stroke rehabilitation is approached. By integrating AR technology with precise sensor data, this system enhances the analysis of complex symptoms associated with brain disorders, thereby improving diagnostic accuracy and therapeutic interventions.
The significance of early detection and continuous observation in managing Parkinson’s
Disease is further emphasized, underscoring the necessity for timely interventions and
personalized care plans to substantially improve life quality for those afected [
In addition to its application in neurodegenerative disease management, this AR system holds potential for transforming post-stroke treatment. By facilitating the monitoring of recovery progress and the efectiveness of rehabilitation strategies, it can significantly contribute to the recovery and quality of life of stroke patients. The system is designed to track a wide array of symptoms and recovery indicators, including but not limited to voice and speech changes, motor function, posture stability, and walking patterns.
Developed using Unity for Microsoft Hololens 2 goggles, the application is well-equipped with multiple sensors, including microphones, cameras, infrared cameras, a depth sensor, and an IMU sensor. This array of technology allows for the comprehensive monitoring of neurological symptoms, ofering a groundbreaking tool for healthcare professionals in diagnosing and managing neurodegenerative diseases and aiding in stroke rehabilitation.
By using the glasses’ cameras to monitor eye movements, the project can assess motor functions and responses to specific tasks presented via AR. Dystonia, which involves slow and sometimes extended intense muscle contractions, is another symptom associated with neurodegenerative diseases. Manifesting through walking dificulties, involuntary movements, and instability of posture when coupled with limb rigidity and spasticity, dystonia’s efects on gait and posture can be detected with the glasses’ accelerometers, gyroscopes, and magnetometers.
Dyskinesias, such as tremors, are indicative of neurodegenerative conditions. Yet, the presence of tremors alone does not sufice for a diagnosis; significant factor here is the slowing of movement (bradykinesia). Tremors are often the first apparent motor symptom and, when other symptoms are minimal, may be the sole basis for an initial diagnosis. The system will analyze tremors and bradykinesia by tracking hand movements using the AR technology. Notably, the design accommodates elderly users, allowing the glasses to be worn over prescription lenses without removal due to their lightweight, wireless design and calibration feature.
This system is crucial for the early detection and ongoing observation of Parkinson’s Disease. By automating diagnostic tasks and utilizing AR technology, it enables clinicians to allocate more time to thorough analysis, enhancing the diagnostic procedure’s eficacy. Early detection allows healthcare providers to initiate timely and tailored treatment plans, significantly benefiting the health and quality of life for those aflicted by Parkinson’s Disease.
Our platform introduces a pioneering component: a self-guided interactive session that lasts 30 minutes and includes 17 distinct activities: • Speech Activities: Participants engage in activities such as describing pictures, retelling stories, producing sustained vowel sounds (“a”, “e”, “i”, “o”, “u”), repeating certain sequences of words (“pataka”, “petaka”, “pakata”), and responding to questions. • Eye Movement Activities: These include tracking rapid eye movements, assessing reflex saccades with latency, dynamic, break, and overlap tests, undertaking voluntary saccades through alternating and pointer-directed tests, observing slow movements and gaze stabilization, and evaluating smooth pursuit movements. • Hand Movement Activities: Tasks involve extending arms forward for a duration of 2 minutes, alternately touching the nose with each hand while keeping the opposite arm extended, resting with hands on knees and observing them, mimicking the action of screwing in light bulbs at eye level, opening and closing fists with extended arms, and conducting a finger touch test. • Assessment of Posture Stability and Walking Patterns: This encompasses the “up and go” test and a walking evaluation.
Spanning across motor skill evaluations, walking exercises, speech challenges, and cognitive tasks, this platform not only precisely captures minute variations in patient’s performances but also integrates these observations with traditional clinical assessments for a comprehensive review. Detailed instructions and outcomes for each activity are provided, enhanced by a calming and instructive narration. Support materials in diverse formats, such as text, audio, and video, ensure a complete understanding of the exercises. Voice commands allow participants to pause, resume, or navigate the activities, increasing interaction and involvement with the system. Figures 1 and 2 provide insights into how patients view instructions for hand movements and interact with AR glasses during the exercises.
This framework is designed to significantly aid in the early detection and continuous
observation of Parkinson’s Disease. By streamlining tasks and applying AR technology, medical
professionals are aforded more time for thorough examination, thus improving diagnostic
precision (Figure 3). Early recognition permits healthcare providers to introduce appropriate
and timely care and treatment plans. Evaluations link the outcomes of these activities to the
patients’ health conditions. This immersive method further includes extensive neurological
evaluations [
The adoption of task automation afords physicians more time to devote to the evaluation of patient performance, eliminating the need to directly oversee test administration. AR glasses are instrumental in reducing patient anxiety and fostering an environment conducive to seamless communication. Our preliminary findings indicate that patients, when engaged in the AR setting, experience a diminishment of the usual test-related stress. This reduction in anxiety leads to a more willing communication, increased motivation, and notably improved performance. The incorporation of 17 varied tasks not only maintains patient interest but also acts as an efective diagnostic instrument, ofering doctors a detailed insight into diferent neurological facets.
In the realm of neurodegenerative diseases, tracking the entirety of symptoms linked to ongoing brain deterioration is essential. This approach is in harmony with the principles of precision medicine, which strives to customize the diagnostic and therapeutic strategies for each individual. The meticulous observation of patients under this system facilitates rapid and informed decisions regarding their care and treatment.
This platform introduces a novel methodology for the diagnosis and oversight of neurodegenerative conditions, with a particular emphasis on Parkinson’s Disease. At its heart, AR technology, through the integration of AR goggles equipped with accurate sensors, establishes an exhaustive framework for capturing and analyzing a spectrum of symptoms associated with the disease.
The project was funded by The National Centre for Research and Development, Poland under Lider Grant no: LIDER/6/0049/L-12/20/NCBIR/2021 and IDUB 500-696-9400/12.