=Paper=
{{Paper
|id=Vol-3742/paper21
|storemode=property
|title=Multi-sensor analysis of cognitive signals for neurological disorders and diseases
|pdfUrl=https://ceur-ws.org/Vol-3742/paper21.pdf
|volume=Vol-3742
|authors=Mykhaylo Petryk,Vitaly Brevus,Mykhaylo Bachynskyi,Andre Pierre Legrand,Mykola Zaiarnyi
|dblpUrl=https://dblp.org/rec/conf/citi2/PetrykBBLZ24
}}
==Multi-sensor analysis of cognitive signals for neurological disorders and diseases==
https://ceur-ws.org/Vol-3742/paper21.pdf
Multi-sensor analysis of cognitive signals for
neurological disorders and diseases
Mykhaylo Petryk1,†,VitalyBrevus1,∗,†Mykhaylo Bachynskyi1, Andre ́ Pierre Legrand2,
and Mykola Zaiarnyi1
Ternopil Ivan Puluj National Technical University, 56 Ruska str., Ternopil 46001, Ukraine
1
Université Paris Sciences et Lettres - ESPCI Paris, France
2
Abstract
The work presents a hardware-software complex for digital diagnostics of neuropsychological
disorders, injuries, and diseases of the human cerebral cortex caused by the consequences of combat
and technogenic injuries, stress in extreme situations, etc. A new methodology and hybrid model
for digital analysis of cognitive neural signals of the human cerebral cortex have been developed,
and a matrix algorithm has been implemented to determine the indicators of the multisensory
impact of EEG signals on the amplitude and frequency characteristics of patient limb tremors. This
allows for rapid and accurate computer diagnosis of neuro-disorders related to combat and
technogenic injuries, detection of damaged areas of the human cerebral cortex, and selection of
effective and timely treatment methods to restore the patient's normal neurological state.
Keywords
tremor, abnormal movements, computer modeling software, multi-sensor cognitive signals
feedback 1
1. Introduction
Computer diagnostic systems use new information technologies to help diagnose and treat
neurological problems and diseases in people who have brain damage from different causes
and injuries, such as stress, extreme situations, combat, or disasters. Manifestations of these
diseases are manifested in humans as abnormal neurological movements (ANM) or
tremors and their extreme forms in the form of critical diseases - Parkinson's and Alzheimer's
diseases [1]. ANM, as unwanted oscillatory movements, refers to the involuntary contraction
of the muscles of a certain part of the body, in particular, the limbs of the hands, eyelids,
organs of speech, etc. [2]. The characteristic features of these ANMs, which lead to a violation
CITI’2024: 2nd International Workshop on Computer Information Technologies in Industry 4.0, June 12–14, 2024,
Ternopil, Ukraine
∗
Corresponding author.
†
These authors contributed equally.
mykhaylo_petryk@tntu.edu.ua (M. Petryk); v_brevus@tntu.edu.ua (V. Brevus); m.bachynskyi@gmail.com (M.
Bachynskyi); andre-pierre.legrand@espci.psl.eu (A. Legrand); mykola.zaiarnyi@gmail.com (M. Zaiarnyi)
0000-0001-6612-7213 (M. Petryk); 0000-0002-7055-9905 (V. Brevus); 0000-0003-4139-7633 (M. Bachynskyi);
7103275600 (AP. Legrand); 0009-0009-0780-4152 (M. Zaiarnyi)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
of the regulation of human movements, are an increase in their amplitude, a change in the
frequency and form of oscillations.
The analysis of these ANM parameters is decisive for understanding the role of feedback
dysfunction of neural nodes of the cerebral cortex in the processes of cognitive control of
human movements and detection of neuromotor disorders. The complexity of ANM research
lies in the imperfection of existing diagnostic methods, their low accuracy, and the lack of
high-quality mathematical and software tools for identifying the neural feedback influences of
cerebral cortex nodes on their behavior [2, 3].
Studies of the behavior of patients with signs of tremor were conducted by a number of
researchers, such as Haubenberg D., Kalowitz D. Legrand A.-P., Vidailhet M., Louis E. and
others [2-5]. The main attention was paid here to the analysis of parameters relative to
normal states and behavior using digital processing methods based on the classical Fourier
transformation [2-4], which are currently imperfect and practically unsuitable in the sense of
a complex analysis of abnormal states and complex, hard-to-predict behavior of patients with
a high degree of tremor [6-7]. Due to the low quality of assessment of ANM parameters of
such methods, 60-80% of important information for assessing the patient's neurological
condition is lost due to noise.
2. Development Methodology
Our proposed method for digital diagnostics of neurological and psychological disorders
relies on information technology. This technology continuously tracks the limb position of a
patient's hand and uses cognitive neuro signals from the cerebral cortex. The cerebral cortex
triggers oscillating movements by responding to these signals while mimicking a specified test
trajectory on a high-resolution interactive tablet using an electronic pen [8]. The electronic
pen's position on the tablet and the sensor readings from the cerebral cortex neuro nodes in
the patient's helmet are recorded synchronously and stored digitally. We analyze this data
using a hybrid model of digital Fourier analysis. This model examines wave signals
representing ANM of the patient's limb, influenced by cognitive signals from the cerebral
cortex neuro nodes. A comprehensive description of this hybrid analysis model appears in our
further writings [8].
2.1. The digital ANM trajectory of a patient's limb movement
We use a model solution previously derived in matrix form [8]. Matrix relations determine
the position of the electronic pen on the interactive tablet. These relations facilitate the
parallelization of calculations.
Figure 1 illustrates the experimental setup for recording EEG signals and ANM movements
using a graphic tablet/monitor. The EEG signals are captured by electrodes attached to the
scalp of the participant, who performs a drawing task on the tablet/monitor. The ANM
movements are recorded by the tablet/monitor as the participant traces a spiral pattern with a
stylus. The EEG signals and ANM movements are synchronized and processed to extract
features related to cognitive load and motor control.
a) EEG signals b) graphic tablet/monitor
Figure 1: Demonstration of using research hardware.
11 t1 , l1 , l1
u1 t1 , l1
21 t1 , l2 , l1 22 t 2 , l 2 , l2
u2 t 2 , l2
...
...
j1 t1 , l j , l1 j 2 t 2 , l j , l2 ... jj t j , l j , l j 0
u
j j j t , l ...
...
n 1,1 t1 , ln 1 , l1 n 1,2 t2 , ln 1 , l2 S t 2 ... n 1, j t j , ln 1 , l j ...
u n 1 t n 1 , ln 1
(1)
n 1,n 1 t n 1 , ln 1 , ln 1
S11 (t1 ) S 21 (t1 ) ... S m1 (t1 ) 1
S12 (t 2 ) S 22 (t 2 ) ... S m2 (t 2 ) 2
... ... ... ... ...
S1 j (t j ) S 2 j (t j ) ... S m j (t j ) i
... ... ... ... ...
S (t ) S (t ) ... S (t )
1n1 n 1 2n 1 n 1 mn1 n 1 m
The following matrices and vectors are used here:
u j (t j , l j ) , j 1, n 1 the vector of amplitude deviations of ANM movements from
standard values (Archimedes spirals), l j , t j — the position of the geometric coordinate z
along the movement trajectory correlates with the Archimedean spiral's linear
transformation. This position is contingent upon the passage of time corresponding to z . The
variable j serves as an index that specifies the sequence number of the elementary segment
within the trajectory of the ANM. Furthermore, n — represents the total count of division
points on the ANM trajectory, segregating it into elementary, more straightforward motion
segments.
ji ti , l j , li , j , i 1, n 1 — the impact matrix, which determines the segmental
feedback effects of cognitive signals on individual elements of ANM, is determined by the
formula [10]:
2
t
ji (ti , l j , li ) 1 cos m i / m / b i V j l j , m Vi m / V l j , m , j ,i 1,n1
2
(2)
m 1
bi
where b i — amplitude characteristic on the i-th segment of ANM, V j l j , m ,
m , m 0, — components of the spectral hybrid Fourier function and the set of spectral
values;
si (t j ) , i 1, m, j 1, n 1 — the matrix of the values of cognitive signals coming from
j
the sensors of the neuron nodes of the cerebral cortex, the number of installed sensors in the
helmet
i , i 1, m — the vector of adaptive coefficients of influence of cognitive signals of the
i-th sensor on indicators of ANM movement elements (determined by the matrix algorithm
presented below).
As a result of matrix calculations, we obtain an analytical vector solution that establishes a
direct connection between the values of amplitude deviations and the influence of the
numerical values of all vectors of cognitive signals throughout the entire duration of ANM:
m
i 1i t1 , l1 , l1 S i t1
u1 t1 , l1 i 1 1
m
2 2 2
i 1
u t , l i 21 t1 , l2 , l1 S i (t1 ) 22 (t 2 , l2 , l2 ) S i (t 2 )
1 2
...
... (3)
m
u j t j , l j i j1 1 j 1 i1 1
( t , l , l ) S ( t ) (t , l
j2 2 j 2 , l ) S i2 2
t ... jj (t ,
j jl , l j ) S ij ( t
j )
i 1
...
...
m m1 (t1 , lm , l1 ) S i1 (t1 ) m 2 (t 2 , lm , l 2 ) S i2 (t 2 ) ... mj (t j , lm , l j ) S i j (t j ) ...
u m t m , l m i
i 1
mm (t m , lm , lm ) S im (t m )
The efficiency and convenience of using vector dependency (4) lie in the ability to
determine specific feedback effects of cognitive signals from all EEG sensors of the system on
each movement segment at the current time ti. This takes into account the current values of
the sensor signals, as well as the aftereffects of the sequence of partially attenuated EEG
sensor signal values at previous time points ti, ti-1,, ..., t1.
2.2. The matrix algorithm for the ANM adaptive coefficients
Considering the specific structure and triangular shape of the matrix of cognitive signals
influence, as well as its multidimensionality compared to the non-square
matrix of cognitive signal sensor indicators, we suggest a method. This method involves
calculating adaptive coefficients for the cerebral cortex sensor's cognitive signals. These
coefficients are computed for distinct groups of m consecutive segments along the ANM
movement trajectory m < n+1:
s1 l1 u1 t1 , l1
1
1 11 l1 s12 l1 s1 j l1
s
... ...
m
2 21 l2
s
s22 l2 ... s2 j l 2 ... s2 l2 u 2 t 2 , l2
...
m
... ...
j s (4)
j1 l j sj 2 l j ... sjj l j ... sj l j u j t j , l j
m
... ...
...
m s lm sm 2 l m smj l j mm lm
s
u m tm , lm
m1 ... ...
Where
s ji (l j ) ji t1 , l j , l1 Si t1 j 2 t2 , l j , l2 Si2 t2 ... jj t j , l j , l j Si j t j ... jm tm , l j , lm Sim tm ,
1
0, j j1
jj1 t j1 , l j , l j1 , j, j1 1, m, i 1, m
0, j j1
Figure 2: The representation of the impact exerted by cognitive signals.
The process of calculating the adaptive coefficients of influence for cognitive signals from
cerebral cortex sensors across various segments of the ANM trajectory can be extended to the
entire set of ANM trajectory segments of dimension n 1 . This generalization enables the use
of a computational scheme that is both rapid and cost-effective compared to machine learning
methods, as depicted in Figure 2.
To increase the accuracy, such calculations can be repeated by shifting the initial positions of
the groups of studied segments of the ANM by several positions up or down in the matrix of
the influence of cognitive signals.
3. Computational analysis of ANM and the cerebral cortex signals.
We can obtain high-quality data on the behavior of the curve drawn by the patient. All
data records are saved and can be analyzed using the developed program, to gain insights into
the patient's neurological condition.
Figure 3: Data cleaning and fitting curve.
Figure 3 shows the user interface of the software for visual analysis of the recorded curve
by providing the csv file containing the recorded data from the tablet. The software displays
the red curve drawn by the patient and the original curve given to it. The software also allows
cleaning some of the noises and fitting the curves.
Figure 4: 3D visualization in Cartesian coordinates for x, y [mm] and t [s].
Figure 4 shows the raw data collected from the tablet, which consists of the x and y
coordinates of the pen tip and the corresponding time stamps. The data is plotted in a 3D
space, where the x-axis represents the horizontal position, the y-axis represents the vertical
position, and the z-axis represents the time. The curve drawn by the patient can be seen as a
trajectory in this 3D space, which reflects the speed, accuracy, and smoothness of the patient's
movement.
In order to obtain the polar coordinates of the pen tip from the Cartesian coordinates, we
need to calculate the radius and the angle of each point in the curve. The radius is given by
the Euclidean distance between the origin.
Figure 5 shows the result of applying this coordinate transformation to the data, where the
x-axis represents the radius and the y-axis represents the angle. The curve in polar
coordinates can reveal the shape and symmetry of the patient's movement, as well as the
variation of the speed and direction. A non-linear least square is applied to smooth the curve
and reduce the noise.
n
( a* , b* , c* , d * ) argmin i f ( i , a, b, c, d ) ,
2
a ,b , c , d i 1 (5)
R fitted f ( , a , b , c , d )
* * * *
n is the number of data points.
i and i are the i th data point.
f (i , a, b, c, d ) is the function we are trying to fit to the data, which is
a b * c * cos( ) d * sin( ) .
a, b, c, and d are the parameters of the function that “curve_fit” (Python 3) is trying to
optimize.
Figure 5: Measured R values and Gaussian filter-based fitting of R data.
Figure 6 shows the deviation of the measured R values from the fitted R values on the y-
axis, and time offset from the beginning of recording on the x-axis.
R R measured R fitted (6)
Figure 6: Residual fluctuation ΔR.
After applying the developed matrix algorithm, the result of the calculations is shown on
Figure 7 as ΔR calculated together with the ΔR measured.
Figure 7: Comparison of ΔR calculated and ΔR measured values for normalized data. MinMax
Scaler for each EEG column separately
Figure 7 shows the comparison of the residual fluctuation ΔR calculated by the matrix
algorithm and the ΔR measured by the EEG device for the normalized data. The data was
normalized using the MinMax Scaler method, which scales each column of the EEG data
separately to the range [0, 1]. The figure displays two scatter plots, one for each condition
(resting and watching), with the x-axis representing the ΔR calculated and the y-axis
representing the ΔR measured.
Figure 8: Comparison of ΔR calculated and ΔR measured values for normalized data. MinMax
Scaler for all EEG columns values range
Figure 8 shows the comparison of the residual fluctuation ΔR calculated by the matrix
algorithm and the ΔR measured by the EEG device for the normalized data. The data was
normalized using the MinMax Scaler method, which scales all the columns of the EEG data
together to the range [0, 1].
Figure 9: Comparison of ΔR calculated and ΔR measured values for normalized data standard
Scaler for each EEG column separately
Figure 9 shows the comparison of the residual fluctuation ΔR calculated by the matrix
algorithm and the ΔR measured by the EEG device for the normalized data. The data was
normalized using the Standard Scaler method, which scales each column of the EEG data
separately to have zero mean and unit variance. The figure illustrates that the two methods of
normalization produce similar results in terms of the correlation between the calculated and
measured values of ΔR. However, the Standard Scaler method seems to have less outliers and
a narrower range of values than the MinMax Scaler method.
Using a high-speed matrix computational procedure, we regularized and enhanced the
adaptive coefficient vectors αi for cognitive signals from all EEG sensors according to
formulas (4) and (2). This resulted in significant model adequacy (over 95%) after several
iterations, specifically 3 regularization iterations. The final coefficient values are presented in
the last column of Table 1.
These data help determine the cognitive impacts of specific neural node zones on the
oscillatory deviations of the patient's limb movement along different sections of the studied
ANR trajectory. The computed αi vector indicators are crucial in this process. Importantly,
this method identifies causal relationships and their quantitative characteristics. It focuses on
the ANR motion zones with the highest oscillatory amplitudes and the αi coefficient values for
cognitive EEG signals of specific neural node zones, as shown in Table 1.
This approach provides neurologists with a key tool to analyze suspicious human cerebral
cortex zones in more detail. It aids in prescribing appropriate specialized restorative therapy,
which aims to improve the condition of individuals with tremors and other anomalies,
including those affected by technological or military factors.
Table 1 shows the results of applying the proposed hybrid model to different neurological
conditions, such as post-traumatic stress disorder, traumatic brain injury, and stroke. The
table compares the descriptive statistic values of values for evaluated normalization
approaches.
Table 1
Descriptive statistics for values
for MinMax Scaler for MinMax Scaler for standard scaler
all EEG channels each EEG channel each EEG channel
Count 5280 5280 5280
Mean 0.644022 1.623022 -3.470627
Std 13947.868361 9269.390746 6504.889147
Min -304957.510678 -147762.812674 -06797.876657
25% -1255.605547 -739.651299 -454.990618
50% -4.297557 -0.476809 0.894929
75% 1263.406560 740.463939 485.236329
max 233558.672605 184031.943394 119814.607694
Therefore, the report demonstrates the effectiveness and efficiency of the proposed hybrid
model for wave signal analysis of abnormal neurological movements.
Conclusions
Utilizing the proposed hybrid model for wave signal analysis of abnormal neurological
movements, influenced by cognitive signals from cerebral cortex neural nodes, the system
facilitates rapid and precise computerized diagnosis of neurological conditions. This diagnosis
stems from combat and man-made injuries. It also identifies the impacted cerebral cortex
regions and quantifies their influence on the overall neurological state of an individual. The
obtained results of such a digital analysis make it possible to determine effective and timely
methods of treatment and restoration of the normal neurological condition of a person.
References
[1] Haubenberger, Dietrich, and Mark Hallett. "Essential tremor." New England Journal of
Medicine 378.19 (2018): 1802-1810.
[2] Legrand A.P., Rivals I., Richard A., Apartis E., Roze E., Vidailhet M., Meunier S., Hainque
E. New insight in spiral drawing analysis methods – Application to action tremor
quantification. J Clinical Neurophysiology, 128 (10) (2017): 1823–1834.
[3] Szumilas, Mateusz, et al. "A multimodal approach to the quantification of kinetic tremor
in Parkinson’s disease." Sensors 20.1 (2019): 184.
[4] J. F. Cavanagh, P. Kumar, A. A. Mueller, S. P. Richardson, and A. Mueen, “Diminished
EEG habituation to novel events effectively classifies Parkinson’s patients,” Clinical
Neurophysiology, vol. 129, no. 2, (Feb. 2018), pp. 409–418. doi:
10.1016/j.clinph.2017.11.023.
[5] Roth, Navit, Orit Braun-Benyamin, and Sara Rosenblum. "Drawing Direction Effect on a
Task’s Performance Characteristics among People with Essential Tremor." Sensors 21.17
(2021): 5814.
[6] Kim, Christine Y., et al. "Repeated spiral drawings in essential tremor: a possible limb-
based measure of motor learning." The Cerebellum 18 (2019): 178-187.
[7] Gil-Martín, Manuel, Juan Manuel Montero, and Rubén San-Segundo. "Parkinson’s disease
detection from drawing movements using convolutional neural networks." Electronics 8.8
(2019): 907.
[8] Petryk M., Gancarczyk T., Khimich O. Methods of Mathematical Modeling and
Identification of Complex Processes and Systems on the basis of High-performance
Calculations (neuro- and nanoporous feedback cyber systems, models with sparse
structure data, parallel computations). Scientific Publishing University of Bielsko-Biala.
Bielsko-Biala, Poland, 2021, 194 p. URL:
https://www.sbc.org.pl/dlibra/publication/584139/edition/549297.