This paper addresses the problem of motion imagery classification from electroencephalogram signals which related with many difficulties such on human state, measurement accuracy, etc. Artificial neural networks are a good tool to solve such kind of problems. Electroencephalogram is time series signals therefore, a Gramian Angular Fields conversion has been applied to convert it into images. GAF conversion was used for classification EEG with Convolutional Neural Network (CNN). GAF images are represented as a Gramian matrix where each element is the trigonometric sum between different time intervals. Grayscale images were applied for recognition to reduce numbers of neural network parameters and increase calculation speed. Images from each measuring channel were connected into one multi-channel image. This article reveals the possible usage GAF conversion of EEG signals to motion imagery recognition, which is beneficial in the applied fields, such as implement it in brain-computer interface GAF transform
Electroencephalography is one of the most popular noninvasive methods for studying brain activity today. Signals of the electroencephalogram (EEG) show the total electrical activity of neurons in the cerebral cortex, studying these data, you can get a lot of useful information about the human condition.
The study of EEG is associated with many difficulties, such as the dependence of signals on age, time of day, the presence of noise, interference and a weak degree of structure.
Classical mathematics methods based on time-frequency,
wave or component analysis can be used to study EEG. However,
their application often does not provide stable results of
recognition of various human conditions, and in some cases their
application becomes extremely difficult due to the complexity of
the algorithms [
The use of artificial neural networks (ANNs) in application
areas such as the brain-computer interface is today a promising
area of research [
There are several approaches to the classification of time
series using ANNs [
The GAF (Gramian Angular Field) method [15] was used to classify EEG signals using convolutional neural networks. In this method, the time series is converted to a polar coordinate system. The matrix G is built on the basis of the data obtained, each element of the matrix is equal to the cosine of the sum of the angles. The resulting matrix is converted into an image, which is fed to the input of the convolutional neural network.
The preservation of the time dependence is ensured by such a transformation. The main diagonal is a special case at k = 0, containing the initial values and angular information.
The GAF matrix is a matrix constructed in a series as follows:
xˆi
1. First, the row is normalized to a segment [
cos(1 1) G cos(2 1) cos(n 1) cos(1 n )
cos( 2 n ) .
cos(n n )
The final matrix stores all the information about the series,
except for the initial boundaries of the values that we lose in step
(1) after the normalization procedure — that is, we can restore
the original series from the matrix obtained, but only scaled to
the interval [
Based on the obtained matrices, images are formed for further use (Fig. 1). The work uses single-color images, since color channels in this case do not carry useful information. This allowed to reduce the number of image channels by 3 times compared with the RGB option. (1) (2) (3) Fig. 1. GAF image example convolutional neural network
In this work, the deep CNN architecture was developed, adapted to classify an EEG signal taking into account the number of measuring electrodes. Input images are fed to the input of the network in the form of a 64-channel image, where each channel is a transformed electroencephalogram signal. The architecture of the deep CNN consists of the main layers: three convolutional and three fully connected. Network parameters were determined by the selection method. 4.
The data presented in [
The data set is EEG signals recorded using the BCI 2000
system [
The order of the experiment and the conversion of the source data into GAF images are presented in Fig. 2. For the training of neural networks, the data of the first subject were used.
Electroencephalograms are signals received from 64 measuring channels. Each signal is converted to a separate image using GAF conversion. The generated images are input to the model. By analogy with the RGB representation, the resulting GAF images are converted into an input vector, however, if 3 channels are used in the case of the RGB image, one for each color, in this case a 64 channel image containing the converted EEG signal is input. The structural scheme of the final model is presented in Fig. 3.
The data of the first subject [17] were used to recognize motor imagery. Images 128 by 128 pixels in size were fed to the input of a neural network. Recognition accuracy was about 97%, with an accuracy of 80% from the data source. 7.
The article examined the possibility of using the GAF conversion method for detecting motor imageries in EEG signals. Without the use of additional filtration, high results can be achieved. The recognition accuracy of motor imageries of movement with the right and left hand and the state of rest was 97% for the studied EEG signals. General results indicate that this method gives higher accuracy than the methods described in the data source. The presented method of classification of electroencephalograms can be used to build a brain-computer interface. 8. 9.
The reported study was funded by RFBR according to the research project № 18-08-00977 А. Machine Vision (ICMV 2017), ICMV Committees, Nov 2017, Vienne, Austria. 10.1117/12.2309486. hal-01743695 [11] R. Östberg, ‘Robustness of a neural network used for image classification : The effect of applying distortions on adversarial examples’, Dissertation, 2018. [12] Qinglong Wang , Wenbo Guo , Kaixuan Zhang , Alexander G. Ororbia, II , Xinyu Xing , Xue Liu , C. Lee Giles, Adversary Resistant Deep Neural Networks with an Application to Malware Detection, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 13-17, 2017, Halifax, NS, Canada [13] Vasilyev A.N., Liburkina S.P., Kaplan A.Y. Lateralization of EEG patterns in humans during motor imagery of arm movements in the brain-computer interface // Zhurnal Vysshei Nervnoi Deyatelnosti Imeni IP Pavlova. 2016. V. 66. № 3. P. 302-312. [14] Wang, Z.; Oates, T. Imaging time-series to improve classification and imputation. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina, 25–31 July 2015; pp. 3939–3945. [15] Wang, Z., Oates, T. Encoding time series as images for visual inspection and classification using tiledconvolutional neural networks, Association for the Advancement of Artificial Intelligence (AAAI) conference,2015. [16] J. Yim and K. Sohn, "Enhancing the Performance of Convolutional Neural Networks on Quality Degraded Datasets," 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, NSW, 2017, pp. 1-8. doi: 10.1109/DICTA.2017.8227427 Nima Hatami, Yann Gavet, Johan Debayle. [17] Zhiguang Wang и Tim Oates. “Spatially Encoding Temporal Correlations to Classify Temporal Data Using Convolutional Neural Networks”. В: CoRR abs/1509.07481 (2015). url: http://arxiv.org/abs/1509.07481.