Acquisition, analysis and classification of EEG signals for control design

Abstract

In the design of brain machine interfaces it is common to use motor imagery which is the mental simulation of a motor act, it consists of acquiring the signals emitted when imagining the movement of different parts of the body. In this paper we propose a machine learning algorithm for the analysis of electroencephalographic signals (EEG) in order to detect body movement intention, combined with the signals issued in a state of relaxation and the state of mathematical activity. That can be applied for brain computer interface (BCI). The algorithm is based on the use of recurrent networks (RRN) and can recognized four tasks which can be used for control of machinery. The performance of the proposed algorithm has an average classification efficiency of 80.13%. This proposed method can be used to translate the motor imagery signals, relaxation activity signals and mathematical activity signals into control signal using a four state to control the directional movement of a drone.

-

The Brain Computer Interfaces (BCIs) are used mostly to help people to restoring some functions when they are severely disabled by a neuromuscular disorder, BCIs also used in healthy people to help them improve their functions [ 2 ]. BCI experiments based on electroencephalogram (EEG) have the advantage of being no-invasive for the subject, besides having no environmental restrictions [ 5 ]. In the case of motor images, brain signals are obtained in most cases, using EEG, due to its ease of use and its high temporal resolution. EEG signals are obtained from multiple channels that are placed on the scalp, which makes the signal more accurate [ 3 ]. Recent studies have shown that EEG-based BCIs allow users to control machines with multiple-state classification. In some studies the electroencephalographic signals have been extracted using imagery motor left, right hand, food or language [ 37, 43–46 ], when listening to English vowels a, i and u [ 39 ], in state of relaxation, read, spell and math activity [ 40 ], imaging certain actions without any physical action, imaging actions without physical movement [ 36, 41 ]. Recently different classification methods have been used, among which are, wavelet transformation [ 36 ], (ANN) Feed forward back-propagation neural network design [ 37, 40 ], recurrent neural networks (RNN) deep neural network (DNN) Adam back-propagation [ 38 ], deep recurrent convolutional neural networks (CNNs) [ 35, 41–45 ]. In this article we propose a new architecture based on Long Short-Term Memory networks (LSTM), this type of recurrent network is used to connect past information with current information, also it is capable of storing a large quantity of information during long periods of time. [ 30 ]. For this purpose, we will use signals obtained from the imagination of movement of the left hand and left foot, state of relaxation and mathematical activity. Additionally, in order to make the system as simple as possible, the EEG signals are extracted from a headset using four EEG channels. 3

Materials and method

In this section we will talk about the materials and methods used for the acquisition of EEG signals, their analysis and classification. 3.1

Experimental protocol

For the acquisition of EEG signals outside the shielded laboratory settings we used an easy to use equipment with few electrodes. The M useT M (EEGheadband created by InteraXon) device 1 is a headband that detects signals from the brain, using circuity technology for detecting EEG. Superficial EEG obtained with the headband is a non-invasive, so it is harmless when acquiring electrical signals emitted by brain neurons, showing brain activity in real time [ 24 ]. The Muse device has four acquisition channels and an android application. For this work 30-second recordings were acquired, 40 of them used the imagery motor of the left-hand movement, 40 used the imagery motor of the left foot, 40 in the state of relapse, and 40 in mathematical activity, for a total of 160 records. The recordings were given in a silent environment without external disturbances. In this experiment the EEG signal is segmented into window frames of 3000 data length, equivalent to 13 seconds. The features are extracted for the four tasks and for only one subject.

Deep learning, CNN and LSTM

Deep neural networks contain layers of superimposed neurons, using more additional hidden layers than a classic artificial neural networks. These additional layers or deep, improves the accuracy of the network. It is possible to extract the features automatically, unlike most of the learning algorithms in which human intervention is required. Each layer trains depending on the output of the previous layer. As it progresses, more complex characteristics are trained [ 50 ].

Recurrent neural networks (RNN) 2 have the capacity to learn characteristics of the data set through time, due to their feedback connections [ 48 ]. RRN uses the recurrent connections to create loops in the neurons of the network, this allows tracking of temporal interactions in the incoming signal.

The processing of the temporal information of the RNN is facilitated, because the network generate patterns that behave according to the value of the previously given pattern, that is, the inclusion of recurrent connections generates a dynamic behavior where the information goes temporarily updating [ 28 ]. Unlike the feed forward neural networks, the RRN have the ability to process random sequences of inputs due to their internal memory. The LSTM is an RRN, that has the ability to learn from the signal, by observing events over long periods of time [ 50 ]. LSTM is a type of recurrent network that unlike other neural networks connects previous information with the current task and learns to use information stored for long periods of time. It is an effective model that is used in sequential data learning problems [ 27 ]. LSTMs are also used to capture longterm temporary dependencies [ 49 ]. The architecture of an LSTM network is a memory cell that has a chain sequence, also maintains its status over time and its non-linear gates regulate the flow of information inside and outside the cell [ 49 ].

The convolutional neural networks (CNN) extract abstract functions to create characteristics in a progressive way by means of convolutional operations, convolutional models usually learn from training of different layers. In each layer CNN extract information or characteristics of the input signal [ 51 ].

Signal acquisition and preprocessing

We used the fourth channel of the Muse device for acquiring the EEG. Then, we select a window of 3000 data per signal, 160 signals in total, 40 per imagery motor of left arm, 40 per imagery motor of left foot, 40 in relapse state and 40 in mathematical activity. Figure 6 shows an EEG signal obtained during this process [ 6 ]. One healthy subject participated in the experiments. During the recordings the subject are instructed not to make overt movements and keep their hands relaxed. The motor imagery task was with close eyes. Each trail is 30 s long, the subject performs four tasks namely, relax, math activity, imagine left arm movement and left foot movement.

– Task 1 – Imaginary motor left foot The subject imagine the movement of the left foot during 30s, without movement. – Task 2 – Imaginary motor left hand The subject imagine the movement of the left hand during 30s, without movement. – Task 3 – Relaxation state The subject do not perform any specific task, but are asked to relax as much as possible and think of nothing in particular during 30s. This task is considered the baseline task for alpha wave production and used as a control measure of the EEG. – Task 4 – Mathematical activity The subject think in mathematical operations, during 30s. 4.2

EEG Recording

EEG is recorded using four gold-plated cup bipolar electrodes placed at the AF7, AF8, TP9 and TP10 locations on the sensorimotor cortex area as per the International 10-20 Electrode Placement System. Figure 4 shows the electrode placement locations. For this experiment were carried out sessions of EEG signal recordings for several days, 40 recordings were obtained for each task, where each recording had a duration of 30 seconds. In which 160 EEG signals were obtained from channels AF7, AF8, TP9 and TP10, whose amplification and sampling is 250Hz. For this experiment, a healthy subject, 30 years old, free of disease or medication, participated. Which avoided blinking the eyes and any other external physical movement. All the information obtained from these electrodes was used in the classification. See the graph of the accuracy during the training 4.

Proposed network architecture

In this research we propose an architecture using a CNN layer with an LSTM layer. We use phyton with Keras library [ 23 ] to code the architecture and process the input data. The architecture can be seen in the figure 5, it consists: – A convolutional layer with 10 filters of size 50. – A LSTM layer with 120 neurons. – Four dense layers of 150, 50, 14, and 4 neurons.

The architectures of CNN and LSTM have allowed modeling temporal information, this was also being used in other works for speech recognition and signal classification [ 51 ]. While the CNN layer allows to optimize the extraction of characteristics of the set of signals and obtain patterns of the characteristics for its classification, an LSTM allows to use the previous information temporarily maintaining the flow of the information inside and outside the network and has a layer called context layer this creates a copy of the hidden layer, with it stores the previous state of the previous pattern [ 49 ]., both allow a more efficient architecture.

The optimization of LSTM parameters is performed sequential manner. First we feed the network with EEG subdivided in windows of 3000 samples. Later we choose the number of epochs and best weight’s initialization on the training set. The optimized parameter values, for LSTM are detailed in 6.

We used a loss function whose objective is try to minimize the loss. It can be the string identifier of an existing loss function such as categorical crossentropy function, when using the categorical crossentropy loss function, your targets should be in categorical format, and one at the corresponding index to the class of the sample.

160 data samples are used in this experiment. The training and testing samples is normalized using categorical normalization algorithm. Selection of the training and testing data is chosen randomly. All four classifiers are trained with 90% data samples and tested with 10% data samples. 6

Results and Discussion

In this section, we can observe the data obtained by the classifier implemented, as well as their percentages of accuracy. 6.1

Classification performance of the Modeled Classifier

In figure 7 we can see the accuracy performance of the architecture proposed in a single subject during training. The classification of the motor imagery signals for the four states is shown in the as the classification obtained from the 160 samples for a subject, and the network was trained with 800 epochs. With the accuracy you can see the performance of CNN with LSTM by time. No artifacts were removed from the EEG data, which demonstrates the robustness of the algorithm. The performance results of the classifier are acceptable with respect to the amount of data used. 80% was obtained in the classification, even though it is necessary to improve the algorithm. Which could be used for the restoration of movement and rehabilitation of people with paraplegics conditions and would allow other people to have direct brain control of external devices in their daily life. The combination of a convolutional network with an LSTM network obtained adequate results during feature extraction and training over long periods of time. The network was able to distinguish between thoughts of imagination and two different states of brain activity. For future research it is proposed to use signals acquired only with movement imagination to proceed to test the classifier. 8

Rodriguez et al.

References

1. R. Ron Angevin, « Retroalimentación en el entrenamiento de una interfaz cerebro computadora usando técnicas basadas en realidad virtual ,» Tesis Doctoral, Universidad de Malaga, p. 256 , 2005 .

2. LaFleur , K. Cassady , A. Doud , K. Shades , E. Rogin y B. He , « Quadcopter control in three-dimensional space using a noninvasive motor imagery-based braincomputer interface , » Journal of neural engineering , vol. 10 , pp. 1 - 16 , 2013 .

3. A. K. Das , S. Surech y N. Sundararajan , « A Robust Interval Type-2 Fuzzy Inference based BCI System,» IEEE , vol. 12 , p. 6 , 2016 .

4. A. K. Das , T. T. Leong , S. Surech y N. Sundararajan, « Meta-cognitive Interval Type-2 Fuzzy Controller for Quadcopter Flight Control-An EEG based Approach ,» IEEE, p. 7 , 2016 .

5. B. Hyung Kim , M. Kim y S. Jo, « Quadcopter flight control using a low-cost hybrid interface with EEG-based classification and eye tracking,» Computers in Biology and Medicine , no 51 , p. 10 , 2014 .

6. X. Mao , M. Li , W. Li , L. Niu , B. Xian , M. Zeng y G. Chen, « Progress in EEG-Based Brain Robot Interaction Systems,» Computational Intelligence and Neuroscience , vol. 2017 , p. 25 , 2017 .

7. L. J. Gómez Figueroa , «Análisis de señales EEG para detección de eventos oculares, musculares y cognitivos,» Trabajo de fin de Máster, p. 121 , 2016 .

8. F. Ramos-Arguelles , G. Morales , S. Egozcue , R. Pabón y M. Alonso, « Basic techniques of electroencephalography: principles and clinical applications,» Scielo Analytics , vol. 32 , p. 14 , 2009 .

9. G. A. Addati y G. Perez Lance , « Introducción a los UAV's, Drones o VANTs de uso civil ,» Econstor, p. 12 , 2014 .

10. M. Gestal Pose , «Introducción a las Redes de Neuronas Artificiales,» p. 20 .

11. G. Parra V , « Procesos Gaussianos MULTI- OUTPUT,» Departamento de Ingeniería Matemática , p. 2 , 2016 .

12. «Pilots Brain Controls Drone,» Professional Engineering, p. 2 , March 2015 .

13. Cochocki , A y Rolf Unbehauen. « Neural networks for optimization and signal processing » John Wiley and Sons, Inc., 1993 .

14. Floreano , D. , and Wood , R. J. ( 2015 ). Science, technology and the future of small autonomous drones . Nature , 521 ( 7553 ), 460 .

15. Lapedes , Alan, y Robert Farber. « Nonlinear signal processing using neural networks: Prediction and system modelling .» 1987 .

16. Hu , Yu Hen, y Jeng-Neng Hwang, eds. « Handbook of neural network signal processing .» ( 2002 ).

17. Yu , F. T. , and Jutamulia , S. ( 1992 ). Optical signal processing, computing, and neural networks . John Wiley & Sons, Inc..

18. Guarnizo Lemus , C. ( 2008 ). Análisis de reducción de ruido en señales EEG orientado al reconocimiento de patrones. Tecno Lógicas , ( 21 ).

19. Zecua , E. , Caballero , I. , Martınez-Carranza , J. , and Reyes , C. A. ( 2016 ). Clasificación de estımulos visuales para control de drones.

20. Yu , Yipeng , et al. « FlyingBuddy2: a brain-controlled assistant for the handicapped . » Ubicomp . 2012 .

21. Hansen , John Paulin, et al. « ”The use of gaze to control drones." Proceedings of the Symposium on Eye Tracking Research and Applications.» ACM , 2014 .

22. Khan , Muhammad Jawad, y Keum-Shik Hong. «hybrid eeg-fnirs-Based eightcommand Decoding for Bci: application to Quadcopter control.» Frontiers in neurorobotics , p. 11 ( 2017 ).

23. http://adventuresinmachinelearning.com/keras-lstm-tutorial/

24. http://developer.choosemuse.com/hardware-firmware/hardware-specifications

25. https://www.parrot.com/global/drones/parrot-bebop-2technicals

26. http://developer.parrot.com/docs/SDK3/

27. Gohritz , K. Knobloch , P.M. Vogt , C. Bonnemann , O. Aszmann . Potential Influence of One-Handedness on Politics and Philosophy of the 20th Century . J.Hand Surg .Am., 2009 , Vol. 34 , 1161 - 1162 .

28. Antona Cortés , C. ( 2017 ). Herramientas modernas en redes neuronales: la librería Keras Bachelor's thesis.

29. Betancourt , O. , Gustavo , A. , Suárez , G. , Franco , B. , Fredy , J. ( 2004 ).

30. Zhang , S. , Yang , Y. , Xiao , J. , Liu , X. , Yang , Y. , Xie , D. , Zhuang , Y. ( 2018 ). Fusing Geometric Features for Skeleton-Based Action Recognition using Multilayer LSTM Networks . IEEE Transactions on Multimedia, X(X) , 1 - 1 . https://doi.org/10.1109/TMM. 2018 .2802648

31. Ron Angevin , R. , « Retroalimentación en el entrenamiento de una interfaz cerebro computadora usando técnicas basadas en realidad virtual ,» Tesis Doctoral, Universidad de Malaga, p. 256 , 2005 .

32. Bonet Cruz , I. , Salazar Martínez , S. , Rodríguez Abed , A. , Grau Ábalo , R. , and García Lorenzo , M. M. ( 2007 ). Redes neuronales recurrentes para el análisis de secuencias . Revista Cubana de Ciencias Informáticas , 1 ( 4 ).

33. Kumar , P. , Saini , R. , Roy , P. P. , Sahu , P. K. , and Dogra , D. P. ( 2018 ). Envisioned speech recognition using EEG sensors . Personal and Ubiquitous Computing , 22 ( 1 ), 185 - 199 .

34. Mohamed , E. A. , Yusoff , M. Z. , Malik , A. S. , Bahloul , M. R. , Adam , D. M. , and Adam , I. K. ( 2018 ). Comparison of EEG signal decomposition methods in classification of motor-imagery BCI . Multimedia Tools and Applications , 1 - 23 .

35. Rahma , O. N. , Hendradi , R. , and Ama , F. ( 2018 ). Classifying Imaginary Hand Movement through Electroencephalograph Signal for Neurorehabilitation . Walailak Journal of Science and Technology (WJST) , 15 ( 12 ).

36. Maksimenko , V. A. , Pavlov , A. , Runnova , A. E. , Nedaivozov , V. , Grubov , V. , Koronovslii , A. , ... and Hramov , A. E. ( 2018 ). Nonlinear analysis of brain activity, associated with motor action and motor imaginary in untrained subjects . Nonlinear Dynamics , 1 - 15 .

37. Szczuko , P. , Lech , M. , and Czyżewski , A. ( 2018 ). Comparison of Classification Methods for EEG Signals of Real and Imaginary Motion . In advances in Feature Selection for Data and Pattern Recognition (pp. 227 - 239 ).

38. Springer, Cham.Tang, X. , Yang , J. , and Wan , H. ( 2018 , April). A Hybrid SAE and CNN Classifier for Motor Imagery EEG Classification . In Computer Science On-line Conference (pp. 265 - 278 ).

39. Springer, Cham. Moinnereau, M. A. , Brienne , T. , Brodeur , S. , Rouat , J. , Whittingstall , K. , and Plourde , E. ( 2018 ). Classification of auditory stimuli from EEG signals with a regulated recurrent neural network reservoir .arXiv preprint arXiv: 1804 .10322.

40. Elakkiya , A. , and Emayavaramban , G. ( 2018 ). Biometric Authentication System Using EEG Brain Signature .

41. Zhang , X. , Yao , L. , Wang , X. , Zhang , W. , Yang , Z. , and Liu, Y. ( 2018 ). Know Your Mind: Adaptive Brain Signal Classification with Reinforced Attentive Convolutional Neural Networks .arXiv preprint arXiv: 1802 .03996.

42. Jiao , Z. , Gao , X. , Wang , Y. , Li , J. , and Xu , H. ( 2018 ). Deep Convolutional Neural Networks for mental load classification based on EEG data . Pattern Recognition , 76 , 582 - 595 .

43. Ozmen , N. G. , and Gumusel , L. ( 2013 , July) . Classification of real and imaginary hand movements for a bci design . InTelecommunications and Signal Processing (TSP) , 2013 36th International Conference on(pp. 607 - 611 ). IEEE.

44. Leuthardt , E. C. , Schalk , G. , Wolpaw , J. R. , Ojemann , J. G. , and Moran , D. W. ( 2004 ). A brain-computer interface using electrocorticographic signals in humans . Journal of neural engineering , 1 ( 2 ), 63 .

45. Zhou , S. M. , Gan , J. Q. , and Sepulveda , F. ( 2008 ). Classifying mental tasks based on features of higher-order statistics from EEG signals in brain-computer interface . Information Sciences , 178 ( 6 ), 1629 - 1640 .

46. Pfurtscheller , G. , Neuper , C. , Schlogl , A. , and Lugger , K. ( 1998 ). Separability of EEG signals recorded during right and left motor imagery using adaptive autoregressive parameters . IEEE transactions on Rehabilitation Engineering , 6 ( 3 ), 316 - 325 .

47. Forney , E. M. , and Anderson , C. W. ( 2011 , July) . Classification of EEG during imagined mental tasks by forecasting with Elman recurrent neural networks . In Neural Networks (IJCNN) , The 2011 International Joint Conference on (pp. 2749 - 2755 ). IEEE.

48. Hema , C. R. , Paulraj , M. P. , Yaacob , S. , Adom , A. H. , and Nagarajan , R. ( 2009 , March). Single trial motor imagery classification for a four state brain machine interface . In Signal Processing and Its Applications , 2009 . CSPA 2009 . 5th International Colloquium on (pp. 39 - 41 ). IEEE.

49. Greff , K. , Srivastava , R. K. , Koutník , J. , Steunebrink , B. R. , and Schmidhuber , J. ( 2017 ). LSTM: A search space odyssey . IEEE transactions on neural networks and learning systems , 28 ( 10 ), 2222 - 2232 .

50. Thomas , J. , Maszczyk , T. , Sinha , N. , Kluge , T. , and Dauwels , J. ( 2017 , October) . Deep learning-based classification for brain-computer interfaces . In Systems, Man, and Cybernetics (SMC) , 2017 IEEE International Conference on (pp. 234 - 239 ). IEEE.

51. Ordóñez , F. J. , Roggen , D. ( 2016 ). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition . Sensors , 16 ( 1 ), 115 .