Electroencephalogram (EEG) is a method of recording brain activity using electrophysiological indicators. Because of its portability and ease of operation, it is widely used in brain computer interface (BCI). Emergency braking intention detection is an effective auxiliary means of man-machine hybrid drive. More and more researchers are interested in decoding the EEG of emergency braking intention. In this paper, a brain power imaging strategy for emergency braking intention detection is proposed. We built a simulated driving environment through the Carla platform. A total of 11 subjects participated in our experiment. We collected EEG of each subject driving a simulated car to complete normal driving and emergency braking tasks, and converted scalp electrode signals into source space signals through EEG source imaging (ESI) technology. We used the activation map of brain regions in the cognitive process of emergency braking to select the period of intention generation, and used three different classifiers to classify different driving conditions. The source imaging method not only improves the detection rate of emergency braking intention, but also advances the prediction time of emergency braking intention.
detection time of emergency braking intention was improved to 300 ms before actual braking [11]. Teng et al. designed an experiment on the sudden appearance of pedestrians on the normal road causing the driver's emergency braking intention [12], and proposed an emergency braking detection method based on EEG power spectrum characteristics and pseudo-online verification. Hernandez et al. studied the feasibility of using driver EEG signals to identify emergency braking intentions when drivers experience cognitive states such as workload, fatigue and stress. In three different cognitive states, the experimental average recognition accuracy of emergency braking intention exceeds 70% [13]. Bi et al. improved the accuracy of detecting the driver's emergency braking intention by combining EEG signals with the car's external environment, and provided a new idea for man-machine driving [14].
As mentioned above, most of the emergency braking intention detection of drivers use traditional classification algorithms to process and analyze signals offline, and the online real-time recognition rate cannot meet the actual application requirements. The EEG signal-to-noise ratio is low and the spatial resolution is low. Moreover the non-invasive acquisition method cannot avoid the volume conduction effect. The study of scalp EEG alone fails to use all the available information of data, especially the information about the location of activity sources in the brain. More scholars have proposed EEG source imaging (ESI) technology. EEG source imaging can describe the temporal and spatial dimensions of brain activity in detail, making it an important and affordable tool for studying the characteristics of the brain and neural network in cognitive and clinical neuroscience [15]. EEG source imaging technology is widely used in different cognitive tasks. Hou et al proposed the combination of ESI and convolutional neural network (CNN) to decode EEG four categories of motor imagination (MI) classification tasks [16]. Rajabioun used the features of brain-derived dynamic visual images extracted from EEG signals to classify four individual moving images [17]. Bai et al performed transient analysis of resting state EEG source space and state transition analysis of patients with disturbance of consciousness (DOC) [18]. They found that different neural coordination patterns, including spatial power patterns, temporal dynamics, spectral shifts, and connectivity structures change on a potentially very fast (millisecond) time scale. Pancholi et al proposed a source aware deep learning framework for hand motion reconstruction based on EEG signals [19]. This work used brain source localization to reliably decode the motion intention, and then used the source space information for channel selection and accurate EEG time period selection. EEG source imaging has become an auxiliary means or an intermediate medium to achieve some cognitive tasks, because the source space signal improves the low spatial resolution of scalp EEG signal and can reflect the activation of brain region.
In this paper, the source space signal is obtained by solving the inverse problem of scalp EEG signal. We use the time series of the source space signal to study the brain activation of the driver's emergency braking intention, and then use the power spectral density of the source space signal as the classification feature to study the detection rate of the driver's emergency braking intention. For the sake of safety, we used a simulation driving platform for experiments. Relevant studies have shown that the simulated driving environment has similar results to the real driving environment in terms of emergency braking intention detection [9]. The driving platform simulation adopts Carla, which is an open urban driving simulator [2]. In the experiment, the subject drove a simulated car and completed a series of emergency braking and normal driving during driving. In case of emergency braking, the subject needs to perform emergency braking immediately according to external clues. EEG signals of subjects were recorded synchronously. Then, we use the EEG source imaging and classification algorithm to identify the braking intention of the subjects prior to actual braking.
The rest of this paper is organized as follows. The methods and materials section describes the theme, experimental setup, data collection, source computation and classification approach. The results section shows the experimental results. Finally, we gave some discussion about the results.
There were 11 subjects, aged 22-36 years with an average age of 25.73 years, including 9 males and 2 females. All subjects were recruited from school volunteers, obtained driving licenses, and had more than 2 years of driving experience. Each subject was right-handed with normal or corrected vision. None of the subjects had a history of psychiatric or other neurological disorders. Before the start of the experiment, the purpose and procedure of the experiment were explained to each subject, and all subjects who participated in our study wrote informed consent in accordance with the Declaration of Helsinki. Before the experiment, the subjects had sufficient sleep (>=8 hours) and did not take any medication within 3 days before the experiment. During the experiment, each subject could end the task at any time without any penalty. If the participants completed the experiment successfully, they were awarded 400 yuan.
As shown in Figure 1, our experimental platform consists of a driving simulator consisting of the
driver's seat, steering wheel, accelerator pedal and brake pedal, an EEG acquisition device
(ActiCHamp, Brain products, Germany) and two computers. The EEG acquisition device will record the
scalp EEG signal of the subject during driving. The application program interface (API) of Carla
driving simulator is used to realize the automatic labeling of EEG signals. The computer has two
functions: (
The experiment includes two situations. One is normal driving, and the other is emergency braking. In case of emergency braking, this setting consists of two virtual cars, namely the front car and the rear car. The front car is controlled by the main test driver, and the speed is kept at 60km/h (the speed can also be changed). The rear car is controlled by the test subject, driving in the same lane and keeping a distance of 6-12m from the front car. The front car decelerates rapidly at random, with the interval between two rapid decelerations ranging from 15 s to 60 s, and reminds the rear car through the brake light. In order to avoid collision, the subject needed to brake immediately when he/she saw the brake light of the front car to begin to flash. In this process, two moments were recorded through the API of the Carla platform. One is the time when the brake light of the front car was on, and the other is the time when the rear car body stepped on the brake pedal. After decelerating for 3 seconds, the front car accelerates again to 60 km/h. The subject continued to drive the car to follow the front car, and maintained a distance of 6 m to 12 m. The distance between the two cars is displayed on the window of the simulation platform, and the subjects can see it in real time. Subjects drove the same car on the same road under emergency braking and broke it spontaneously and randomly every 15-60 seconds. After stepping on the brake pedal for about 3 s, the subject accelerated the car to follow the front car and kept the distance between 6-12 m. Each subject needs to complete 5 driving tasks, and each task lasts for 30 minutes. After each driving task, the subjects had a 5 minutes rest, as shown in Figure.2.
In this experiment, the German ActiCHamp amplifier and its electrodes were used to record the EEG signals of the subjects. The internationally recognized 10-20 system is adopted for electrode arrangement, in which 28 electrodes (F5, F3, FZ, F4, F6, FT7, FC5, FC1, FC2, FC6, FT8, T7, C3, CZ, C4, T8, Cp5, CP1, CP2, CP6, P5, P3, PZ, P4, P6, O1, O2 and O2) are used for recording data. Two electrodes (TP9 and TP10) are used as reference electrodes, and 1 electrode (FPZ) is used as grounding electrode, as shown in Figure 3. The resistance of all electrodes shall be kept below 10 kΩ. EEG data acquisition sampling rate is set to 200 Hz. In this paper, EEG data are simply preprocessed. First, FIR band-pass filters with low frequency 1 Hz and high frequency 45 Hz are used to filter the original signals. Then signals with amplitudes greater than 300 µ v were eliminated. The EEG signals correspond to two situations: normal driving and emergency braking. In case of emergency braking, the time when the subject starts to brake and then starts to step on the brake pedal is zero. The target time range that we choose is -3000 ms to 1000 ms. The normal driving range is extracted through a sliding window (4000 ms long) on the EEG signals, which is at least 3000 ms away from the braking behavior of any subject.
The processing of EEG forward and inverse problems is very important for source computation. We made a model to describe the composition, shape distribution and electrical conductivity of brain tissue. The boundary element method (BEM) [20] is used to solve the EEG forward problem, generate a leading field, and convert the activities in the sensor space into those in the source space. Then we use the minimum norm estimation (MNE) [16] method to calculate the source. The lead field and scalp EEG signals were taken as input, and the estimated cortical potential was taken as output. Finally, we get the time series of the activation source point.
We choose three commonly used machine learning classification algorithms: support vector machine (SVM), K-nearest-neighbor (KNN) classifier and adaptive boosting (AdaBoost) algorithm to detect emergency braking intention. The three classifiers have been widely used in EEG decoding and have good classification performance. For normal driving and emergency braking, data from 2000 ms to 1000 ms before emergency braking are selected through the sliding window with the window size of 1 s. The power spectral density of the time series of the source point activated in the source space is used as the feature input classifier. We used 80% of the trials of each subject as the training set and the rest as the test set through 5-fold cross validation. Finally, we used the prediction time and its accuracy rate to evaluate the performance on the test set.
Figure 4 shows the EEG topographic map of every 100 ms interval from 1000 ms before braking to the beginning of braking. Figure 4(a) and 4(b) represent normal driving and emergency braking, respectively. In normal driving, we observed EEG potential with small fluctuation. In contrast, the electrode potential of the occipital area changes slightly 700 ms before the emergency braking, and the electrode potential of the parietal lobe region changes significantly at 400 ms before the braking, indicating that emergency braking activates the cognitive process of the brain, which makes it possible to detect emergency braking. Figure 5 shows the activation map of the brain area in the source space with the interval of every 100 ms. It can be clearly seen that the occipital area responsible for processing visual information is activated at 600 ms before braking. While the parietal lobe area which is responsible for movement is activated at 400 ms before braking. Moreover, the closer it is to the braking time, the stronger the activation is. It can be seen that the intention of the brain to generate emergency braking become obvious about at 400 ms before braking.
In order to detect the intention of emergency braking, we selected 400 ms, 300 ms and 200 ms data before braking for classification. Figure 6 shows the average classification accuracy of support vector machine, nearest neighbor and adaptive enhancement for 11 subjects' data before and after source tracing respectively. The results show that the source imaging method improves the classification accuracy of emergency braking and normal driving about 1% to 5% for different classifiers. Especailly, for the 200 ms, 300 ms and 400 ms before the start of braking, the classification accuracy of the source imaging method is improved more significantly than those of the traditional method. These results show that the source imaging method of EEG has the advantage in predicting the time to detect the intention.
Figure 7 shows the classification results of the three classifiers at different times. The Figure 7(a) and 7(b) show that the source imaging method at before-braking 300 ms can achieve the comparable classification accuracy of the models without source imaging at before-braking 200 ms. Figure 7(c) show that the classification accuracy of the source imaging method at before-braking 400 ms is also significantly improved. These results demonstrate that under the same classification accuracy requirements, the source imaging method can identify the emergency braking intention in advance and reduce the prediction of emergency braking time. (a) (b) Figure 4 (a) EEG topographic map of normal driving. (b) EEG topographic map of emergency braking.
Classification accuracy of three classifiers before and after source imaging at different (c) Figure 7 Classification results of subject 3. (a) Ada (b) KNN (c) SVM
In this study, we proposed ESI method to deal with the driver's emergency braking data in the man-machine hybrid driving. For the time series of active source points in the source space, the power spectral density is extracted and then input to three different classifiers. The results show that the combination of ESI and machine learning classification algorithm can improve the detection rate of emergency braking intention and advance the prediction time of emergency braking intention.
However, in the near future, we will optimize source computing and combine ESI with other advanced classification algorithms to obtain higher intention detection rate and earlier intention prediction time, which can be applied to online detection of emergency braking intention. 5 6
This work is supported by National Key Research and Development Program
(No.2018YFB1305101) and National Defense Science and Technology Innovation Special Zone
Program (KY0103052116).
[14] Bi L, Wang H, Teng T, et al. A novel method of emergency situation detection for a
braincontrolled car by combining EEG signals with surrounding information[J]. IEEE Transactions on
Neural Systems and Rehabilitation Engineering, 2018, 26(
Processing and Control, 2020, 61: 102028. [18] Bai Y, He J, Xia X, et al. Spontaneous transient brain states in EEG source space in disorders of consciousness[J]. NeuroImage, 2021, 240: 118407. [19] Pancholi S , Giri A , Jain A , et al. Source Aware Deep Learning Framework for Hand Kinematic
Reconstruction using EEG Signal[J]. 2021.
[20] Ilić S S, Perić M T, Aleksić S R, et al. Hybrid boundary element method and quasi-TEM analysis
of two-dimensional transmission lines—generalization[J]. Electromagnetics, 2013, 33(