Mathematics and software for controlling mobile software devices based on brain activity signals Oleh Pastukh1, Volodymyr Stefanyshyn1, Ihor Baran1, Ihor Yakymenko2, Vasyl Vasylkiv1 1 Ternopil Ivan Puluj National Technical University, 56, Ruska Street, Room 1-104 Ternopil, 46001, Ukraine 2 West Ukrainian National University, 11 Lvivska str., Ternopil, 46009, Ukraine Abstract This article explores innovative approaches to processing cognitive neurosignals originating from the cerebral cortex, focusing on the analysis of digital data acquired from EEG sensors, utilizing advanced machine learning techniques. The primary goal of this research is to create a sophisticated framework for modeling the movements of limbs, with a specific emphasis on identifying distinct types of limb movement. In particular, this investigation delves into the reverse cognitive effects associated with the analysis of individual movement elements, specifically examining the thumbs of both hands. The proposed methodologies show significant potential in advancing bionic prostheses, enabling more intuitive and precise control of artificial limbs through the interpretation of brain signals. Keywords 1 Neuroprosthetics, biomechanical simulations, brain-machine interaction, brain activity signals. 1. Introduction Machine learning methods based on neural network technologies have achieved a high level of accuracy and productivity when used in computer data (signal) processing systems. These breakthroughs have greatly enhanced our capacity to tackle challenges associated with recognizing and identifying human movements influenced by cognitive signals originating from nodes within the cerebral cortex. This technological progress holds profound implications for numerous medical applications, including the restoration of motor functions in individuals affected by various traumatic events, such as accidents or injuries related to military service. Furthermore, it extends to the treatment of patients grappling with debilitating neurological pathologies, Parkinson's diseases [1-5]. A critical aspect of comprehending and effectively restoring human movements hinges on the analysis of digital signals originating from cerebral cortex (CC) nodes. The intricate interplay between cognitive influences from these nodes and their impact on motor control mechanisms underscores the importance of precise signal analysis. However, existing diagnostic methods are far from perfect, often characterized by limited precision and a shortage of mathematical and software tools designed to discern the intricate interplay of neurophysiological influence on the part of individual neuronal groups on motor behavior. Notably, various researchers, have delved into neural system studies that seek to decipher patient behavior [2-5]. Their efforts have primarily centered on assessing the state of motor support mechanisms (MSM) in patients, utilizing classical digital processing methods founded on techniques such as Fourier transformation [6-9]. In this article, we harness cutting-edge advancements in the field of machine learning to delve into the analysis of data acquired from electroencephalograms (EEG). Our primary aim is to explore and refine methods that can discern neural signals associated with various human movements. By Proceedings ITTAP’2023: 3rd International Workshop on Information Technologies: Theoretical and Applied Problems, November 22–24, 2023, Ternopil, Ukraine, Opole, Poland EMAIL: oleg.pastuh@gmail.com (A. 1); volodymyr_stefanyshyn3006@tntu.edu.ua (A. 2), Ihor.remm@gmail.com (A. 3), jiz@wunu.edu.ua (A. 4); vasylkivv@gmail.com (A. 5) ч: 0000-0002-0080-7053 (A. 1); 0009-0007-2829-8995 (A. 2); 0000-0002-8153-2476 (A. 3); 0000-0001-6956-5753 (A. 4); 0000-0001-8517- 3223 (A. 5) ©️ 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings leveraging sophisticated machine learning techniques, particularly those rooted in deep neural networks, we aim to identify the most effective approaches for recognizing models of EEG signals of the activity of the cerebral cortex, which provide movements of the thumbs of the left and right hands. This research endeavor is driven by the overarching goal of contributing to the development of high- precision prosthetic devices[10-13]. We aspire to bridge the gap between neural signals and prosthetic limb control, ultimately enhancing the quality of life for individuals who have experienced limb loss. Moreover, this investigation holds the potential to pave the way for future advancements in the field, opening up new possibilities for innovative prosthetic technologies. Our exploration into this realm not only seeks to enhance our understanding of neural processes but also aims to empower individuals who have faced limb amputations with more intuitive and accurate control over their artificial limbs. By harnessing the power of contemporary machine learning techniques, we aspire to unlock new dimensions in the analysis of cognitive signals, ultimately leading to the development of prosthetic solutions that can greatly improve the daily lives of those with limb impairments. The significance of this research extends beyond the present moment, offering a glimpse into the possibilities and opportunities that lie ahead in the realm of prosthetic technology and neural signal processing [9-18]. 2. EEG Data Acquisition In the pursuit of our research objectives, we designed an experiment to acquire essential EEG data related to specific elementary finger movements. To ensure the integrity and reliability of the data, a controlled environment was meticulously prepared for the experiment. The participant was situated in a dedicated room, carefully chosen to minimize external factors that could potentially influence the patient's neural signals. This controlled environment was essential for capturing clean and precise data. For the experiment, we employed a state-of-the-art electroencephalograph (EEG) equipped with 16 sensors strategically positioned to capture neural activity from distinct regions of the brain. The patient was comfortably seated in a chair, and measures were taken to eliminate any sources of interference, including ambient light and external auditory stimuli. This ensured that the data collected during the experiment would be devoid of external disturbances. During the course of the experiment, the participant was instructed to perform a simple task: flex and extend the fingers of one hand and then repeat the same movements with the other hand. These deliberate finger movements were chosen as they represent fundamental motor actions, making them an ideal subject for our investigation. The data collection phase lasted for a duration of four minutes, with neural signals recorded at a frequency of 250 Hz (Fig 1) [14]. As a result of this carefully executed experiment, we obtained a substantial dataset consisting of 6003 data points for each hand. These EEG measurements serve as the foundation for our subsequent analysis, where we endeavor to unlock insights into the cognitive feedback effects of specific finger movements, furthering our understanding of neural processes associated with motor control. (Fig 2-3). The deliberate elimination of external factors that could potentially influence the patient's neural signals was a crucial aspect of our experiment. By meticulously crafting an environment devoid of external interferences, we ensured the high quality and integrity of the data we collected. This controlled setting minimized any unintended variables that could confound our analysis, ultimately allowing us to obtain EEG data of exceptional quality. The absence of external factors played a pivotal role in the success of our research, enabling us to delve into the intricate nuances of neural processes associated with specific finger movements with a high degree of precision and confidence. Figure 1: Encephalographic Data from 16 Neural Channels: A Comprehensive View of Brain Activity Figure 2: Input Data from the Encephalograph for the first hand Figure 3: Input Data from the Encephalograph for the second hand 3. Processing EEG Signal Data We used the Python for data processing and analysis, which provided us with a versatile set of tools. These libraries played a crucial role in efficiently handling the data and analyzing the results. 3.1. Initial Data Processing In the subsequent stage of our experiment, the focus was on preparing the acquired data for our machine learning model. Following the experiment, we obtained two distinct datasets, namely "eeg_right_finger.txt" and "eeg_left_finger.txt." Each of these files contained data regarding brain signals associated with finger movements. These dataset comprised 17 columns, with the first column containing information about the measurement time ("time"), and the remaining 16 columns representing signal values recorded at that time, labeled as "Fp1," "Fp2," "F3," and so on (see Fig. 5 and 6). Since both datasets were recorded at the same frequency, we only needed the data sequences corresponding to the brain signal values for further processing [14]. To manage and manipulate the data, we utilized the functionalities provided by the pandas library. For each of the datasets, we introduced an additional field that would serve as an identifier for associating the signals with the respective hand's movements. We designated the movements of the left hand's fingers with the identifier "0" and correspondingly used "1" for the movements of the right hand's fingers. This identifier was named "target." An illustration of the resulting table can be seen in Figure 4. To facilitate data import and data selection, we employed the "read" method for reading the data and the "iloc" method to remove extraneous columns that were not pertinent to our analysis. Figure 4: Input data with finger index The dataset from both data frames can be observed in Figures 5 and 6. These figures display the distribution of values for all data entries of brain signal measurements for both hand movements. This visualization helps provide a clearer view of the distribution of sensor values. We can observe the distribution throughout the entire duration of the experiment (4 minutes). Figure 5: First data set for “target:0” Figure 6: Second data set for “target:0” 3.2 Multilayer Perceptron for Data Analysis Furthermore, for the analysis of data, we will employ a multilayer perceptron (MLP) to develop our machine learning model, allowing for accurate classification of finger movements based on the EEG signals recorded during the experiment. Figure 7: Multilayer Perceptron (MLP) Architecture In the data preparation phase, our primary objective was to filter out irrelevant information and create a clean dataset for training our model. To achieve this, we merged the data from both LHF and RHF sets into a single dataset, effectively doubling our training examples. This merger allowed us to simulate finger movements from both hands randomly. After merging the datasets, we conducted data cleaning by removing any incorrect signals and tuples containing missing values. This data quality control step was crucial to ensure the integrity of our dataset. Next, we standardized the signal values within the dataset to have a range from 0 to 1. Standardization was essential for ensuring that all signals carried equal weight during model training. This standardization process contributes to better model convergence and faster training. During the model development phase, we experimented with various machine learning models to define the best approach for our task. One key component of our model was the utilization of a multilayer perceptron (MLP) to classify finger movements based on EEG signals. Subsequently, we divided our preprocessed data into two sets in a 3:1 ratio. The larger portion served as the training data for teaching our system, while the smaller portion acted as validators for fine-tuning our model's performance [14]. It's essential to emphasize that this meticulous data preprocessing and model selection approach were driven by the necessity for high accuracy in recognizing specific limb movements when applied to bionic prosthetics. Achieving precise movement recognition is paramount for enhancing the functionality and usability of such prosthetic devices. 3.3 Neural Network Training and Validation With all the preparatory work completed, we proceeded to initiate deep neural network. The first step involve model initialization and training. Following the training phase, it is crucial to test the model's accuracy in predicting outcomes. Thanks to the standardization and normalization of our data, the training process is significantly expedited. We employed the backpropagation of error process, enabling our model. As a result, we obtained the following performance metrics: • f1_score = 0.999221, • accuracy_score = 0.999211, • roc_auc_score = 0.999204. The determination of accuracy with these results directly influences the quality of future prosthetic usage. To achieve this level of accuracy, we utilized various evaluation techniques. This high level of accuracy ensures the correct interpretation of complex limb movements based on brain signal data, paving the way for precise control of prosthetic devices in the future [14]. 3.4 Practical application of Machine Learning Algorithm In the course of our work, we trained the deep neural network using dataset obtained from EEG measurements, specifically signals from the brain (Figure 8): Figure 8: Input dataset for recognizing movements. Our developed network accurately identifies the corresponding limb movement with an impressive precision of approximately 99.9%. These results pave the way for further exploration of more complex movements, and our research findings serve as a robust foundation for these endeavors. Concluisions Our research introduces a robust information technology framework designed for the analysis of digital EEG signals originating from the central cortex (CC). This technology serves as a powerful tool for investigating the status of the human motor system, leveraging the capabilities of machine learning, neural networks. Our innovative approach enables the precise identification of individual movement elements, taking into account the cognitive feedback loops within the CC neural nodes, which serve as distinctive features for recognizing of limb movements. One of the notable outcomes of this study is the development of high-performance algorithms tailored for the recognition of movement elements. These algorithms have demonstrated the potential for parallel computing, enhancing efficiency and scalability. The implications of our research extend beyond the confines of the laboratory. The knowledge and insights gained from this experiment hold significant promise for advancing the field of bionic prosthetics. By better understanding how cognitive feedback influences human limb movements, we are poised to enhance the design and functionality of bionic prosthetic devices. Our work has the value of improving the quality of life of people who rely on such prostheses by offering them greater control and precision in their daily activities. In conclusion, this research paves the way for more effective and sophisticated bionic prosthetic solutions, bringing us closer to a future where these devices seamlessly integrate with the human body, restoring mobility and autonomy to those in need. References [1] Bhidayasiri R., Mari Z. Digital phenotyping in Parkinson's disease: Empowering neurologists for measurement-based care. Parkinsonism Relat Disord. 2020 Nov; 80:35-40. DOI: 10.1016/j.parkreldis.2020.08.038. [2] Legrand A.P., Rivals I., Richard A., Apartis E., Roze E., Vidailhet M., Meunier S., Hainque E. New insight in spiral drawing analysis methods – Application to action tremor quantification. J Clinical Neurophysiology, 128 (10), 1823–1834. (2017) [3] Pancholi, S., Joshi, A.M. Advanced Energy Kernel-Based Feature Extraction Scheme for Improved EMG-PR-Based Prosthesis Control Against Force Variation. IEEE Transactions on Cybernetics, 52(5), pp. 3819–3828. (2023) [4] Borra D., Mondini V., Magosso E., Müller-Putz G.R. Decoding movement kinematics from EEG using an interpretable convolutional neural network. Computers in Biology and Medicine, 165, 107323. [5] Petryk M., Gancarczyk T., Khimich O. Methods of Mathematical Modeling and Identification of Complex Processes and Systems on the basis of High-performance Calculations (neuro- and nanoporous feedback cyber systems, models with sparse structure data, parallel computations). Scientific Publishing University of Bielsko-Biala. Bielsko-Biala, Poland, 2021, 194 p.https://www.sbc.org.pl/dlibra/publication/584139/edition/549297 [6] Mudryk I., Petryk M.. High-performance modeling, identification and analysis of heterogeneous abnormal neurological movement’s parameters based on cognitive neuro feedback-influences. Innovative Solutions in Modern Science. 2021. New York. TK Meganom LLC. V1(45), 235-249 doi: 10.26886/2414-634X.1(45)2021.16. [7] Petryk M., Vorobiev E. Liquid Flowing from Porous particles During the Pressing of Biological Materials. Computer and [8] Petryk M., Khimitch A., Petryk M.M., Fraissard J. Experimental and computer simulation studies of dehydration on microporous adsorbent of natural gas used as motor fuel. Fuel. Vol. 239, 1324–1330 (2019) [9] Petryk M., Leclerc S., Canet D., Sergienko I., Deineka V., Fraissard J. Competitive Diffusion of Gases in a Zeolite Bed: NMR and Slice Selection Procedure, Modelling and Parameter Identification. The Journal of Physical Chemistry C. ACS (USA). Vol. 119. Issue 47, 26519–26525 (2015). [10] Petryk M.R., Boyko I.V., Khimich O.M., Petryk O.Y. High-Performance Methods of Modeling the Adsorption with Feedback in Heterogeneous Multicomponent NanoporousMedia. Cybernetics and System Analysis, Springer New York, Vol. 58(5), 787- 805 (2022) DOI 10.1007/s10559-022-00512-8 [11] Lebovka N., Petyk M., Tatochenko M. and Vygornitskii N. Two-stage random sequential adsorption of discorectangles and disks on a two-dimensional surface. Physical Review E. Vol.108, 024109 (2023) DOI: https://doi.org/10.1103/PhysRevE.108.024109 [12] Petryk M., Boyko I., Fessard J., Lebovka N. Modelling of non-isothermal adsorption of gases in nanoporous adsorbent based on Langmuir equilibrium. Adsorption. Vol. 29, 141–150 (2023) DOI https://doi.org/10.1007/s10450-023-00389-9 [13] Lebovka N., Petyk M., Vorobiev E. Monte Carlo simulation of dead-end diafiltration of bidispersed particle suspensions. Physical Review E. Vol.106. 064610 (2022) DOI 10.1103/PhysRevE.106.064610 [14] Petryk M., Pastukh O., Bachynskyi M., Mudryk I., Stefanyshyn V. Processing of Cerebral Cortex Neurosignals from EEG Sensors and Recognizing Specific Types of Mechanical Movements Elements of Pacient Limbs under the Cognitive Feedback Influenses. СІТІ-2023. Ternopil, Ukraine, June 14-16, 2023, Session 1: Day 1, 61-70 [15] Petryk M.R., Khimich A., Petryk M.M., Fraissard J. Experimental and computer simulation studies of dehydration on microporous adsorbent of natural gas used as motor fuel, 2019. Fuel 239, pp. 1324-1330.