Multi-sensor analysis of cognitive signals for neurological disorders and diseases Mykhaylo Petryk1,†,VitalyBrevus1,∗,†Mykhaylo Bachynskyi1, Andre ́ Pierre Legrand2, and Mykola Zaiarnyi1 Ternopil Ivan Puluj National Technical University, 56 Ruska str., Ternopil 46001, Ukraine 1 Université Paris Sciences et Lettres - ESPCI Paris, France 2 Abstract The work presents a hardware-software complex for digital diagnostics of neuropsychological disorders, injuries, and diseases of the human cerebral cortex caused by the consequences of combat and technogenic injuries, stress in extreme situations, etc. A new methodology and hybrid model for digital analysis of cognitive neural signals of the human cerebral cortex have been developed, and a matrix algorithm has been implemented to determine the indicators of the multisensory impact of EEG signals on the amplitude and frequency characteristics of patient limb tremors. This allows for rapid and accurate computer diagnosis of neuro-disorders related to combat and technogenic injuries, detection of damaged areas of the human cerebral cortex, and selection of effective and timely treatment methods to restore the patient's normal neurological state. Keywords tremor, abnormal movements, computer modeling software, multi-sensor cognitive signals feedback 1 1. Introduction Computer diagnostic systems use new information technologies to help diagnose and treat neurological problems and diseases in people who have brain damage from different causes and injuries, such as stress, extreme situations, combat, or disasters. Manifestations of these diseases are manifested in humans as abnormal neurological movements (ANM) or tremors and their extreme forms in the form of critical diseases - Parkinson's and Alzheimer's diseases [1]. ANM, as unwanted oscillatory movements, refers to the involuntary contraction of the muscles of a certain part of the body, in particular, the limbs of the hands, eyelids, organs of speech, etc. [2]. The characteristic features of these ANMs, which lead to a violation CITI’2024: 2nd International Workshop on Computer Information Technologies in Industry 4.0, June 12–14, 2024, Ternopil, Ukraine ∗ Corresponding author. † These authors contributed equally. mykhaylo_petryk@tntu.edu.ua (M. Petryk); v_brevus@tntu.edu.ua (V. Brevus); m.bachynskyi@gmail.com (M. Bachynskyi); andre-pierre.legrand@espci.psl.eu (A. Legrand); mykola.zaiarnyi@gmail.com (M. Zaiarnyi) 0000-0001-6612-7213 (M. Petryk); 0000-0002-7055-9905 (V. Brevus); 0000-0003-4139-7633 (M. Bachynskyi); 7103275600 (AP. Legrand); 0009-0009-0780-4152 (M. Zaiarnyi) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings of the regulation of human movements, are an increase in their amplitude, a change in the frequency and form of oscillations. The analysis of these ANM parameters is decisive for understanding the role of feedback dysfunction of neural nodes of the cerebral cortex in the processes of cognitive control of human movements and detection of neuromotor disorders. The complexity of ANM research lies in the imperfection of existing diagnostic methods, their low accuracy, and the lack of high-quality mathematical and software tools for identifying the neural feedback influences of cerebral cortex nodes on their behavior [2, 3]. Studies of the behavior of patients with signs of tremor were conducted by a number of researchers, such as Haubenberg D., Kalowitz D. Legrand A.-P., Vidailhet M., Louis E. and others [2-5]. The main attention was paid here to the analysis of parameters relative to normal states and behavior using digital processing methods based on the classical Fourier transformation [2-4], which are currently imperfect and practically unsuitable in the sense of a complex analysis of abnormal states and complex, hard-to-predict behavior of patients with a high degree of tremor [6-7]. Due to the low quality of assessment of ANM parameters of such methods, 60-80% of important information for assessing the patient's neurological condition is lost due to noise. 2. Development Methodology Our proposed method for digital diagnostics of neurological and psychological disorders relies on information technology. This technology continuously tracks the limb position of a patient's hand and uses cognitive neuro signals from the cerebral cortex. The cerebral cortex triggers oscillating movements by responding to these signals while mimicking a specified test trajectory on a high-resolution interactive tablet using an electronic pen [8]. The electronic pen's position on the tablet and the sensor readings from the cerebral cortex neuro nodes in the patient's helmet are recorded synchronously and stored digitally. We analyze this data using a hybrid model of digital Fourier analysis. This model examines wave signals representing ANM of the patient's limb, influenced by cognitive signals from the cerebral cortex neuro nodes. A comprehensive description of this hybrid analysis model appears in our further writings [8]. 2.1. The digital ANM trajectory of a patient's limb movement We use a model solution previously derived in matrix form [8]. Matrix relations determine the position of the electronic pen on the interactive tablet. These relations facilitate the parallelization of calculations. Figure 1 illustrates the experimental setup for recording EEG signals and ANM movements using a graphic tablet/monitor. The EEG signals are captured by electrodes attached to the scalp of the participant, who performs a drawing task on the tablet/monitor. The ANM movements are recorded by the tablet/monitor as the participant traces a spiral pattern with a stylus. The EEG signals and ANM movements are synchronized and processed to extract features related to cognitive load and motor control. a) EEG signals b) graphic tablet/monitor Figure 1: Demonstration of using research hardware.  11  t1 , l1 , l1    u1  t1 , l1         21  t1 , l2 , l1   22  t 2 , l 2 , l2    u2  t 2 , l2        ...   ...         j1  t1 , l j , l1   j 2  t 2 , l j , l2  ...  jj  t j , l j , l j  0  u  j j j  t , l    ...   ...        n 1,1  t1 , ln 1 , l1   n 1,2  t2 , ln 1 , l2  S  t 2  ...  n 1, j  t j , ln 1 , l j  ...  u n 1  t n 1 , ln 1       (1)  n 1,n 1  t n 1 , ln 1 , ln 1    S11 (t1 ) S 21 (t1 ) ... S m1 (t1 )   1       S12 (t 2 ) S 22 (t 2 ) ... S m2 (t 2 )   2   ... ... ... ...   ...        S1 j (t j ) S 2 j (t j ) ... S m j (t j )    i       ... ... ... ...   ...   S (t ) S (t ) ... S (t )     1n1 n 1 2n 1 n 1 mn1 n 1   m  The following matrices and vectors are used here: u j (t j , l j )  , j  1, n  1 the vector of amplitude deviations of ANM movements from standard values (Archimedes spirals), l j , t j — the position of the geometric coordinate z along the movement trajectory correlates with the Archimedean spiral's linear transformation. This position is contingent upon the passage of time corresponding to z . The variable j serves as an index that specifies the sequence number of the elementary segment within the trajectory of the ANM. Furthermore, n — represents the total count of division points on the ANM trajectory, segregating it into elementary, more straightforward motion segments.  ji  ti , l j , li  , j , i  1, n  1 — the impact matrix, which determines the segmental   feedback effects of cognitive signals on individual elements of ANM, is determined by the formula [10]:   2    t   ji (ti , l j , li )    1  cos  m i   /   m / b i  V j  l j ,  m Vi   m  / V  l j ,  m  , j ,i 1,n1 2 (2) m 1    bi    where b i — amplitude characteristic on the i-th segment of ANM, V j  l j ,  m  ,  m , m  0,  — components of the spectral hybrid Fourier function and the set of spectral values;  si (t j )  , i  1, m, j  1, n  1 — the matrix of the values of cognitive signals coming from  j  the sensors of the neuron nodes of the cerebral cortex, the number of installed sensors in the helmet  i  , i  1, m — the vector of adaptive coefficients of influence of cognitive signals of the i-th sensor on indicators of ANM movement elements (determined by the matrix algorithm presented below). As a result of matrix calculations, we obtain an analytical vector solution that establishes a direct connection between the values of amplitude deviations and the influence of the numerical values of all vectors of cognitive signals throughout the entire duration of ANM: m       i 1i  t1 , l1 , l1  S i  t1   u1  t1 , l1    i 1 1     m  2 2 2   i 1  u  t , l      i  21  t1 , l2 , l1  S i (t1 )   22 (t 2 , l2 , l2 ) S i (t 2 ) 1 2       ...  ...    (3)   m   u j  t j , l j     i j1 1 j 1 i1 1  ( t , l , l )  S ( t )   (t , l j2 2 j 2 , l )  S i2  2  t  ...   jj (t , j jl , l j )  S ij ( t  j )      i 1    ...  ...       m   m1 (t1 , lm , l1 ) S i1 (t1 )   m 2 (t 2 , lm , l 2 ) S i2 (t 2 )  ...   mj (t j , lm , l j ) S i j (t j )  ...   u m  t m , l m      i      i 1     mm (t m , lm , lm ) S im (t m )   The efficiency and convenience of using vector dependency (4) lie in the ability to determine specific feedback effects of cognitive signals from all EEG sensors of the system on each movement segment at the current time ti. This takes into account the current values of the sensor signals, as well as the aftereffects of the sequence of partially attenuated EEG sensor signal values at previous time points ti, ti-1,, ..., t1. 2.2. The matrix algorithm for the ANM adaptive coefficients Considering the specific structure and triangular shape of the matrix of cognitive signals influence, as well as its multidimensionality compared to the non-square matrix of cognitive signal sensor indicators, we suggest a method. This method involves calculating adaptive coefficients for the cerebral cortex sensor's cognitive signals. These coefficients are computed for distinct groups of m consecutive segments along the ANM movement trajectory m < n+1:  s1  l1   u1  t1 , l1   1 1    11  l1   s12  l1   s1 j  l1  s ... ...    m     2    21  l2  s  s22  l2  ...  s2 j  l 2  ...  s2  l2   u 2  t 2 , l2   ...   m       ...  ...   j    s     (4)    j1  l j   sj 2  l j  ...  sjj  l j  ...  sj  l j   u j  t j , l j      m ...   ...     ...     m    s  lm   sm 2  l m   smj  l j   mm  lm  s   u m  tm , lm      m1 ... ... Where s ji (l j )   ji  t1 , l j , l1  Si  t1    j 2  t2 , l j , l2  Si2  t2   ...   jj  t j , l j , l j  Si j  t j   ...   jm  tm , l j , lm  Sim  tm  , 1  0, j  j1    jj1 t j1 , l j , l j1   , j, j1  1, m, i  1, m  0, j  j1 Figure 2: The representation of the impact exerted by cognitive signals. The process of calculating the adaptive coefficients of influence for cognitive signals from cerebral cortex sensors across various segments of the ANM trajectory can be extended to the entire set of ANM trajectory segments of dimension n  1 . This generalization enables the use of a computational scheme that is both rapid and cost-effective compared to machine learning methods, as depicted in Figure 2. To increase the accuracy, such calculations can be repeated by shifting the initial positions of the groups of studied segments of the ANM by several positions up or down in the matrix of the influence of cognitive signals. 3. Computational analysis of ANM and the cerebral cortex signals. We can obtain high-quality data on the behavior of the curve drawn by the patient. All data records are saved and can be analyzed using the developed program, to gain insights into the patient's neurological condition. Figure 3: Data cleaning and fitting curve. Figure 3 shows the user interface of the software for visual analysis of the recorded curve by providing the csv file containing the recorded data from the tablet. The software displays the red curve drawn by the patient and the original curve given to it. The software also allows cleaning some of the noises and fitting the curves. Figure 4: 3D visualization in Cartesian coordinates for x, y [mm] and t [s]. Figure 4 shows the raw data collected from the tablet, which consists of the x and y coordinates of the pen tip and the corresponding time stamps. The data is plotted in a 3D space, where the x-axis represents the horizontal position, the y-axis represents the vertical position, and the z-axis represents the time. The curve drawn by the patient can be seen as a trajectory in this 3D space, which reflects the speed, accuracy, and smoothness of the patient's movement. In order to obtain the polar coordinates of the pen tip from the Cartesian coordinates, we need to calculate the radius and the angle of each point in the curve. The radius is given by the Euclidean distance between the origin. Figure 5 shows the result of applying this coordinate transformation to the data, where the x-axis represents the radius and the y-axis represents the angle. The curve in polar coordinates can reveal the shape and symmetry of the patient's movement, as well as the variation of the speed and direction. A non-linear least square is applied to smooth the curve and reduce the noise. n ( a* , b* , c* , d * )  argmin   i  f ( i , a, b, c, d )  , 2 a ,b , c , d i 1 (5) R fitted  f ( , a , b , c , d ) * * * * n is the number of data points. i and  i are the i  th data point. f (i , a, b, c, d ) is the function we are trying to fit to the data, which is a  b *  c * cos( )  d * sin( ) . a, b, c, and d are the parameters of the function that “curve_fit” (Python 3) is trying to optimize. Figure 5: Measured R values and Gaussian filter-based fitting of R data. Figure 6 shows the deviation of the measured R values from the fitted R values on the y- axis, and time offset from the beginning of recording on the x-axis. R  R measured  R fitted (6) Figure 6: Residual fluctuation ΔR. After applying the developed matrix algorithm, the result of the calculations is shown on Figure 7 as ΔR calculated together with the ΔR measured. Figure 7: Comparison of ΔR calculated and ΔR measured values for normalized data. MinMax Scaler for each EEG column separately Figure 7 shows the comparison of the residual fluctuation ΔR calculated by the matrix algorithm and the ΔR measured by the EEG device for the normalized data. The data was normalized using the MinMax Scaler method, which scales each column of the EEG data separately to the range [0, 1]. The figure displays two scatter plots, one for each condition (resting and watching), with the x-axis representing the ΔR calculated and the y-axis representing the ΔR measured. Figure 8: Comparison of ΔR calculated and ΔR measured values for normalized data. MinMax Scaler for all EEG columns values range Figure 8 shows the comparison of the residual fluctuation ΔR calculated by the matrix algorithm and the ΔR measured by the EEG device for the normalized data. The data was normalized using the MinMax Scaler method, which scales all the columns of the EEG data together to the range [0, 1]. Figure 9: Comparison of ΔR calculated and ΔR measured values for normalized data standard Scaler for each EEG column separately Figure 9 shows the comparison of the residual fluctuation ΔR calculated by the matrix algorithm and the ΔR measured by the EEG device for the normalized data. The data was normalized using the Standard Scaler method, which scales each column of the EEG data separately to have zero mean and unit variance. The figure illustrates that the two methods of normalization produce similar results in terms of the correlation between the calculated and measured values of ΔR. However, the Standard Scaler method seems to have less outliers and a narrower range of values than the MinMax Scaler method. Using a high-speed matrix computational procedure, we regularized and enhanced the adaptive coefficient vectors αi for cognitive signals from all EEG sensors according to formulas (4) and (2). This resulted in significant model adequacy (over 95%) after several iterations, specifically 3 regularization iterations. The final coefficient values are presented in the last column of Table 1. These data help determine the cognitive impacts of specific neural node zones on the oscillatory deviations of the patient's limb movement along different sections of the studied ANR trajectory. The computed αi vector indicators are crucial in this process. Importantly, this method identifies causal relationships and their quantitative characteristics. It focuses on the ANR motion zones with the highest oscillatory amplitudes and the αi coefficient values for cognitive EEG signals of specific neural node zones, as shown in Table 1. This approach provides neurologists with a key tool to analyze suspicious human cerebral cortex zones in more detail. It aids in prescribing appropriate specialized restorative therapy, which aims to improve the condition of individuals with tremors and other anomalies, including those affected by technological or military factors. Table 1 shows the results of applying the proposed hybrid model to different neurological conditions, such as post-traumatic stress disorder, traumatic brain injury, and stroke. The table compares the descriptive statistic values of values for evaluated normalization approaches. Table 1 Descriptive statistics for values for MinMax Scaler for MinMax Scaler for standard scaler all EEG channels each EEG channel each EEG channel Count 5280 5280 5280 Mean 0.644022 1.623022 -3.470627 Std 13947.868361 9269.390746 6504.889147 Min -304957.510678 -147762.812674 -06797.876657 25% -1255.605547 -739.651299 -454.990618 50% -4.297557 -0.476809 0.894929 75% 1263.406560 740.463939 485.236329 max 233558.672605 184031.943394 119814.607694 Therefore, the report demonstrates the effectiveness and efficiency of the proposed hybrid model for wave signal analysis of abnormal neurological movements. Conclusions Utilizing the proposed hybrid model for wave signal analysis of abnormal neurological movements, influenced by cognitive signals from cerebral cortex neural nodes, the system facilitates rapid and precise computerized diagnosis of neurological conditions. This diagnosis stems from combat and man-made injuries. It also identifies the impacted cerebral cortex regions and quantifies their influence on the overall neurological state of an individual. The obtained results of such a digital analysis make it possible to determine effective and timely methods of treatment and restoration of the normal neurological condition of a person. References [1] Haubenberger, Dietrich, and Mark Hallett. "Essential tremor." New England Journal of Medicine 378.19 (2018): 1802-1810. [2] Legrand A.P., Rivals I., Richard A., Apartis E., Roze E., Vidailhet M., Meunier S., Hainque E. New insight in spiral drawing analysis methods – Application to action tremor quantification. J Clinical Neurophysiology, 128 (10) (2017): 1823–1834. [3] Szumilas, Mateusz, et al. "A multimodal approach to the quantification of kinetic tremor in Parkinson’s disease." Sensors 20.1 (2019): 184. [4] J. F. Cavanagh, P. Kumar, A. A. Mueller, S. P. Richardson, and A. Mueen, “Diminished EEG habituation to novel events effectively classifies Parkinson’s patients,” Clinical Neurophysiology, vol. 129, no. 2, (Feb. 2018), pp. 409–418. doi: 10.1016/j.clinph.2017.11.023. [5] Roth, Navit, Orit Braun-Benyamin, and Sara Rosenblum. "Drawing Direction Effect on a Task’s Performance Characteristics among People with Essential Tremor." Sensors 21.17 (2021): 5814. [6] Kim, Christine Y., et al. "Repeated spiral drawings in essential tremor: a possible limb- based measure of motor learning." The Cerebellum 18 (2019): 178-187. [7] Gil-Martín, Manuel, Juan Manuel Montero, and Rubén San-Segundo. "Parkinson’s disease detection from drawing movements using convolutional neural networks." Electronics 8.8 (2019): 907. [8] Petryk M., Gancarczyk T., Khimich O. Methods of Mathematical Modeling and Identification of Complex Processes and Systems on the basis of High-performance Calculations (neuro- and nanoporous feedback cyber systems, models with sparse structure data, parallel computations). Scientific Publishing University of Bielsko-Biala. Bielsko-Biala, Poland, 2021, 194 p. URL: https://www.sbc.org.pl/dlibra/publication/584139/edition/549297.