Sentiment Analysis: Using Detrended Fluctation Analysis of EEG Signals in Natural Reading

Abstract

While Natural Language Processing (NLP) techniques can be used to identify sentiment in text, information sources such as the neural signals of a reader are typically not incorporated into the process. In this paper, we investigated whether measures extracted from Electroencephalography (EEG) signals during reading could be used to identify the sentiment of sentences. Our study used the ZuCo dataset which contained 18 channels of EEG collected from 10 native English speakers as they read 400 sentences. Each sentence belonged to a positive, negative or neutral sentiment class. We show how Detrended Fluctuation Analysis (DFA), an extension to chaotic systems fluctuation analysis, can be used to identify diferences and changes in human EEG for reading texts with diferent sentiments. Based on DFA, on each time scale, we found that the left and right occipital electrodes had the greatest activation between sentiment conditions, and the EEG at electrodes over temporalfrontal scalp sites showed a significant change over many frequency bands for texts of diferent sentiment. Additionally, we also compared DFA to descriptive statistics to show that DFA is a useful technique for EEG analysis.

Keywords

• EEG
• DFA
• sentiment
• statistics
• electrode

Introduction

Sentiment analysis is a branch of Natural Language Processing (NLP) that is used to indicate and understand opinions expressed in written language. One area where this is frequently used is on the reviews that can be found in many common publicly available datasets e.g. Stanford Sentiment Treebank[ 18 ], Amazon product data, IMDB Movie Reviews Dataset, Twitter US Airline Sentiment[ 3 ].

However, there are many challenges when using sentiment analysis approaches.

Many questions have been raised regarding how human language works when people directly read sentences or documents that contain sentiment.

The next stage in the development of NLP will involve the incorporation of diferent modalities instead of concentrating only on text-based data [ 10,1 ]. In recent years, using EEG (Electroencephalography) signals in linguistic research, especially in normal reading has enabled numerous insights [ 11,7,9,19 ]. However, Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) although combining neural signals with artificial intelligence methods may seem tempting, it is still a challenging task because the signals often contain noise, artifacts, or interference [ 16 ]. In order to achieve a better understanding of how to use EEG signals in natural reading tasks, and to combine them with NLP methods to get closer to human-level language comprehension, we need novel approaches to combine EEG (Electroencephalography) signals with sentiment analysis. Previous studies have reported that EEG signals are nonlinear, selfafine, and non-stationary [ 4,22 ]. Thus, traditional statistical methods of EEG analysis (e.g. Fourier transform) may fail to capture important properties of the signals since these techniques are intended for stationary signals with independent frequencies [ 12 ]. In this sense, Detrended Fluctuation Analysis (DFA) has been shown to be useful for non-stationary time series [ 5 ], especially brain activity signals.

There are many studies that have successfully applied DFA for EEG signal analysis. Zebende et al[ 20 ] applied DFA to analyze four diferent channels from a total of 64 with three time scales corresponding to frequencies smaller than 40Hz in the motor/imaginary human task. Similarly, Filho et al.[ 13 ] used DFA to analyze 22 channels of EEG in a reading task performed by one subject that had been trained before an experiment while the other participants were not.

They also achieved clear diferences in 11 channels between two subjects by using DFA for four frequency (f) bands including δ , θ , α , and β (f < 30Hz).

In this paper we used the Zurich Cognitive Language Processing Corpus (ZuCo) dataset [ 7 ]. This is a publicly available dataset that provides simultaneous EEG and eye-tracking data for natural sentence reading to support aspectbased sentiment analysis. In the original paper for this dataset, they consider EEG signals based on four frequency bands including γ , β , α , and θ . Other studies that used this dataset have typically focused on decoding and combining EEG signals with text-based data to address the sentiment analysis task [ 6,14 ].

At present there is no study that identifies which EEG electrode locations might be important for sentiment analysis. In order to answer this question, we applied both the traditional technique (descriptive statistics) and DFA. Along with the main question, with the DFA method, we can observe the changes of each electrode and compare them to each other for specific frequencies that cannot be explained by the descriptive statistics.

The rest of the paper is structured as follows. First, we provide a dataset description and describe the data preprocessing. This is followed by the methodology that describes the procedures of using descriptive statistics and DFA.

Following this, Section 4 illustrates main results and analysis. The conclusion is then presented in Section 5. 2

Dataset and Processing

The ZuCo dataset1 was first released for reading tasks in this paper [ 7 ]. It consists of EEG measures co-registered to eye fixations for participants reading text. All 1 https://osf.io/q3zws/ participants were presented with 400 sentences describing movie reviews that belonged to one of three types of sentiment categories i.e. positive, negative, and neutral. These sentences in the ZuCo dataset were selected randomly from the Stanford Sentiment Treebank (SST) [ 18 ]. The SST has been used in the task of sentiment analysis in which algorithms must analyze and predict the sentiment of a sentence. There were 123 negative, 137 neutral, and 140 positive sentences used for subjects in the ZuCo dataset, accounting for approximately 4% of the SST dataset. For the sentiment task, a control pad was used to switch to the next sentence and the subjects had to indicate the correct sentiment. Unlike Rapid Serial Visual Representation (RSVP) [ 8 ] in which each word is displayed independently at a uniform speed, the experimental setup ensured natural reading behaviours. This means each of the subjects could read each sentence at various speeds and could freely reread words or even the entire sentence.

During normal reading, the perceived sentiment of text can be afected by surroundings, domain knowledge or interests. The ZuCo dataset was recorded in a controlled manner in order to mitigate these influences. Similarly, domain knowledge or interests of the subject can impact their perceived sentiment of text i.e. if people read a text and its content is outside their domain of expertise or is not a topic of interest for them, the sentiment they choose for the sentence could be influenced. The sentences that were used in the ZuCo dataset experiment were selected from the Stanford Sentiment Treebank, a corpus with fully labeled parse trees from movie reviews that should be suitable for everyone. Besides, the average number of seconds a subject spends per sentence is around 5.52. With the reading speed and the content of movie reviews, reader’s sentiment associated with their domain knowledge or interests should not have a significant efect i.e. the sentiment they choose for each sentence.

The ZuCo dataset includes both raw and preprocessed EEG data using Automagic (version: 1.4.6)2. There are 105 channels of EEG, with an additional 9 channels that are used for EOG artifact removal, as well as other channels corresponding to locations on the neck and face that were excluded from the dataset due to noise. The procedure for EEG preprocessing using Automagic is described in [ 7 ]. In this study, we extracted the final data from the preprocessing step and built a pipeline to get 105 channels of EEG that were band-pass filtered between a frequency range from 4 Hz to 50 Hz. Given that EEG channels that correspond to nearby electrode sites tend to be correlated, not all of the 105 channels were analysed. If we analyzed all 105 channels, this would be cumbersome for analysis and visualization. Hence we chose a representative selection of electrodes. [ 17 ], they used 8 channels Fp1, Fp2, F7, F8, T3, T4, F3 and F4 according to the 10-20 system to implement the sentiment classification task. Another paper also assessed EEG signals for emotion recognition but they implemented their experiment on visual-based data [ 21 ]. However, in terms of emotion recognition, similar to our task, they concluded that FP2, O2, T7, T8 channels contained sig2 The whole preprocessing can be found at https://github.com/methlabUZH/ automagic nificant activity related to sentiment recognition. Most previous studies use the 10-20 International System for electrode placement, however, the ZuCo dataset was collected on a 128-channel EEG Geodesic Hydrocel system that does not perfectly align with this electrode placement system. Electrode Cz was used as a reference during data recording, where after the experiment the data was rereferenced to a common average reference. In this paper, we use 18 channels of EEG with nearly the same locations as the International 10-20 System (shown in Figure 1 for convenience of analysis and interpretation. These electrodes are symmetrical with respect to the midline on the scalp covering regions above the left and right frontal lobes, parietal lobes, occipital lobes, temporal lobes, and central region.

(a) (b)

The average time for each sentiment sentence is approximately 5.52 seconds while the sentence that accounted for the longest time to read was nearly 14.8 seconds and the shortest one was around 1.1 seconds. For each sentiment sentence, we used EEG signals recorded from 19 electrodes for 10 subjects. Therefore, we had 23,370, 26,030, and 26,600 epochs according to each sentiment condition. 3

Methodology

3.1

Statistical techniques

There are two types of statistical techniques that were used to analyse EEG signals during the sentiment reading task. Firstly, descriptive statistics were applied in order to have an overview of all signals from collected channels via statistical measures such as mean, standard deviation, skewness, and kurtosis. Also, DFA is used to find diferences between any two considered channels for three groups of sentiment sentences. According to Peng’s work [ 15 ], mathematically DFA can be decomposed and implemented in the following steps: Step 1: Calculate the average cumulative sum y(k) of an EEG signal (xi), with i = 1, 2, ..., N − 1, N , and N the number of samples. From this definition, we achieve the integrated series y(k) = Pk i=1(xi− x), where x is the mean value and k = 1, 2, ..., N − 1, N .

Step 2: Subdivide the averaged cumulative sum yk in n samples and for each subsample compute a least square linear fitting yn(k). Repeat this process at diferent scales, namely from dividing the signal in many n time scales. Step 3: Compute the detrended fluctuation F (n) as the root-mean-square deviation between the averaged cumulative sum y(k) and the fitting yn(k), at each scale:

v F (n) = tuu N1

N X(y(k) − yn(k))2

(1) k=1 Step 4: Express ∆logF DF A as a log function with the number of n samples to compare two signals.

EEG signals can be analysed in terms of their power spectral density, amplitude, shape, and position. Power spectral density is a fundamental method to determine the diferences in rhythms. Hence, for this analysis, we analyzes the EEG signals in terms of four diferent frequency bands according to common analysis frequency bands as follows: θ (4-8Hz), α (8.5-13Hz), β (13.5-30Hz), and γ (30.5-49.5Hz). Based on these frequency bands and sampling rate of 500Hz (∆t = 0.002), four ranges of window size from 10 to 125 corresponding to time scale from 0.02s to 0.25s are used for the DFA method. This is detailed in Table 1.

If the FDF A value of a channel is higher than other channels for a particular time scale, we refer to that channel as being more active. If we compare FDF A values between two or more channels, comparing FDF A directly is not suitable as it will fail to show diferences due to the small diferences between channels. Thus, the ∆logF DF A function is used as this method has been shown to be successful by other authors[ 20 ] in the analysis of the time series of motor/imaginary EEG. Where Fd is FDF A value of the most active channel and Fx is FDF A value of electrode x, the formula for this function becomes: ∆logF

DF A := logFd − logFx (2)

DF A, the relationship between two electrodes can be analyzed as i: If ∆logF DF A > 0, the diference in the log-amplitude of DFA between the most active channel and electrode x is larger. ii: If ∆logF DF A = 0, the diference in the log-amplitude of DFA between the most active channel and electrode x is zero. iii: If ∆logF DF A < 0, the diference in the log-amplitude of DFA between the most active channel and electrode x is smaller. 4 4.1

Results and Analysis

Descriptive statistics

We calculated statistical measures of entire sentiment sentences for all subjects at the sentence level for 18 channels of EEG. In Figure 2, time-series EEG at each electrode is considered in three sentiment conditions, and summarised by mean, standard deviation, skewness, and kurtosis. The calculated statistical measures do not show clear diferences between negative (blue circle), neutral(orange circle), and positive (green circle) in the EEG for sentiment conditions when analysed across all subjects. A one-way ANOVA was used to evaluate whether there is any diference between three sentiment conditions across all channels for each subject with 95% confidence intervals. P-values for mean, standard deviation, skewness, and kurtosis were 0.070, 0.995, 0.901, and 0.169 respectively. Thus, there was no statistically significant diference in all statistical measures according to three sentiment conditions (p − values > 0.05). With regard to the mean measure, all channels approach zero. For skewness measures, since all values for 18 channels were between -0.5 and 0.5, they were fairly symmetrical. Kurtosis tells us the tails of distributions of the considered signals. According to the kurtosis plot, all values are smaller than 3, which means the distributions are platykurtic i.e. the distribution is shorter, tails are thinner than the normal distribution. The low kurtosis values also indicate that all electrodes had a lack of outliers thus being amenable for analysis. E70 had the highest standard deviation in three classes. 4.2

Detrended Fluctuation Analysis

To begin, we use a one-way ANOVA to test whether there is any significant diference between FDF A of each electrode over ten subjects in terms of three sentiment conditions. The distribution of p values is 0.55 ± 0.27. Since our p values are all larger than 0.05, we do not have a statistically significant diference between sentiment conditions on a per participant-electrode combination. Therefore, we can compute FDF A values without taking subjects into account.

The results of the FDF A values potentially reflect the discrimination between diferent electrode signals. In raw data, we cannot determine which one has the greatest amplitude over others. However, based on FDF A, for most of the time scales, the FDF A value of E70 is always higher than that of other channels. This indicates that E70 is the most active channel. Considering all time scales, the FDF A values of E70 and E83 are not much diferent for each condition. Therefore, we need to implement the next step to calculate ∆logF DF A to identify the time scale having the greatest diference between E70 and E83 as well as other electrodes.

All computed ∆logF DF A are illustrated in Figure 3. In this figure, there are four clusters that depend on the distance from electrode positions to the occipital electrodes since we calculate the diference between E70 (left of occipital lobe) and others. That also explains the reason why ∆logF DF A becomes smaller if an electrode channel is near the electrode E70. Based on the conditions of ∆logF DF A, for ∆logF DF A > 0, we find that the E70 competes with E83 as the most active channel in a half of β band frequency and in both γ while E83 replaces this for the time scale n > 30 or frequency f < 16.67Hz (∆logF DF A < 0). Additionally, ∆logF DF A values of E33 (red line) and E122 (brown line) at electrodes at temporal-frontal scalp sites are significantly diferent for the 3 sentiment conditions. Therefore, we visualize ∆logF DF A between two channels separately to observe their characteristics.

Looking at Figure 4, ∆logF DF A < 0 since the low β wave with f < 16Hz, E83 has become the most active channel instead of E70 in this case. However, what stands out from this figure is that negative sentences (blue) appear to change in the most active channel when frequency is larger (n is smaller) than others whereas this alternative occurs for both neutral (orange) and positive (green) at the same frequency. According to the graph, it is dificult to determine the diference between three conditions represented by three time series. In other to verify these diferences, a one-way ANOVA is used, where we find out there is a significant diference in the ∆logF DF A of both channels in α (p − value = 0.0075 < 0.05) and θ (p − value = 0.0026 < 0.05) frequencies for the 3 sentiment conditions.

Figure 5 indicates the changes in ∆logF DF A of the E33 and the E122 for three conditions over a number of time scales. Generally, we observed that the value of ∆logF DF A are apparently diferent. Using ∆logF DF A which is a method to show the diference in logFDF A of two electrodes is actually efective in this case. Moreover, we also used a one-way ANOVA to test a confidence interval for this diference. P-values of all band frequency for three types of conditions were 2.10− 14, 6.2.10− 21, 6.8.10− 24, and 5.7.10− 19 for γ , β , α , and θ that were all smaller than 0.05, indicating we have evidence to prove that there is a diference in logFDF A between both channels. Considering ∆logF DF A between both channels, in term of γ band frequency (n between 10 and 16) and time scale from a part of the high β wave with 20Hz < f < 25Hz (16 < n < 20), for the negative label, the E122 is more active than the E33 because ∆logF DF A < 0 while the E33 is more active for both neutral and positive sentences (∆logF DF A > 0). For the time scale n between 20 and 35 (25Hz < f < 14Hz), the E122 is more active than the E33 for the negative and positive classes since ∆logF DF A is smaller then zero while the E33 is more active than others for the neutral label. During the considered time scale n from 35 to 125 (θ and α waves), because ∆logF DF A < 0 in all sentences, the E122 is more active than the E33. In this paper we analyzed brain activity patterns using DFA for aspect-based sentiment analysis during reading. The dataset used consisted of 10 native English speakers with EEG captured as they read reviews with diferent sentiments.

Our analysis concentrated on three types of sentiment sentences, namely neutral, positive, and negative. Using descriptive statistics, we did not find any diference in EEG signals for participant-electrode combination in terms of negative, positive, and neutral sentences. We applied DFA methods and found diferences and changes in 18 channels. In terms of electrode signals, the E70 channel corresponding to an electrode placed at a left occipital scalp site, and the E83 channel corresponding to an electrode placed at a right occipital scalp site, had the highest DFA values. Moreover, we observed that there was a significant difference in the ∆logF DF A of the E33 and the E122 for electrodes placed at left and right temporal-frontal scalp sites for the 3 sentiment conditions.

For each frequency band, for the aforementioned channels, the majority of the changes occurred in the beta frequency band. In this band, E83 was the most active channel instead of E70, and E122 was more active than E33. However, every change happened in diferent types of beta waves (high or low beta band) that depended on each condition. Both neutral and positive labels had relatively similar behaviours for most of the channels. Nevertheless, there was a change in E122 for E33 starting to appear in neutral and positive groups were the low beta and the high beta waves, respectively.

There is scope to apply the analyses presented in this paper in aspect-based sentiment analysis for natural reading tasks. Currently, there are two main approaches to use EEG signals with machine learning. The first one can be conceived as a bottom up method that begins with engineered features and then uses these features to build a suitable model to predict the reading task [ 21 ]. Another approach is decoding EEG signals to feed them into Deep Learning models along with text-based data to improve the final accuracy. If we choose the former, we need to use time series techniques or statistical methods to obtain suitable features. For the latter, this approach is currently favoured by many researchers in the NLP domain[ 2,14,6 ]. However, it can be dificult to understand and explain the importance of electrode signals when a deep learning approach directly learns the features in tandem with text data. Hence, our analysis, which is the first step in a feature engineering approach, gives insight on useful features present in EEG that could support sentiment prediction of text based on ∆logF DF A for each frequency band, and also help researchers to reduce the number of used channels to decode EEG signals.

Acknowledgements This publication has emanated from research conducted with the financial support of Science Foundation Ireland under Grant number 18/CRT/6183. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.

References

1. Bisk , Y. , Holtzman , A. , Thomason , J. , Andreas , J. , Bengio , Y. , Chai , J. , Lapata , M. , Lazaridou , A. , May , J. , Nisnevich , A. , et al.: Experience grounds language . arXiv preprint arXiv: 2004 . 10151 ( 2020 )

2. Brennan , J.R. , Hale , J.T. : Hierarchical structure guides rapid linguistic predictions during naturalistic listening . PloS one 14(1) , e0207741 ( 2019 )

3. Gamal , D. , Alfonse , M. , M El-Horbaty , E.S. , M Salem , A.B. : Analysis of machine learning algorithms for opinion mining in diferent domains . Machine Learning and Knowledge Extraction 1 ( 1 ), 224 - 234 ( 2019 )

4. Gao , J. , Hu , J. , Tung , W.w.: Complexity measures of brain wave dynamics . Cognitive neurodynamics 5 ( 2 ), 171 - 182 ( 2011 )

5. Hardstone , R. , Poil , S.S. , Schiavone , G. , Jansen , R. , Nikulin , V.V. , Mansvelder , H.D. , Linkenkaer-Hansen , K. : Detrended fluctuation analysis: a scale-free view on neuronal oscillations . Frontiers in physiology 3 , 450 ( 2012 )

6. Hollenstein , N. , Renggli , C. , Glaus , B. , Barrett , M. , Troendle , M. , Langer , N. , Zhang , C. : Decoding eeg brain activity for multi-modal natural language processing . arXiv preprint arXiv:2102.08655 ( 2021 )

7. Hollenstein , N. , Rotsztejn , J. , Troendle , M. , Pedroni , A. , Zhang , C. , Langer , N.: Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading . Scientific data 5(1) , 1 - 13 ( 2018 )

8. Lees , S. , McCullagh , P. , Payne , P. , Maguire , L. , Lotte , F. , Coyle , D. : Speed of rapid serial visual presentation of pictures, numbers and words afects event-related potential-based detection accuracy . IEEE Transactions on Neural Systems and Rehabilitation Engineering 28 ( 1 ), 113 - 122 ( 2019 )

9. Ling , S. , Lee , A.C. , Armstrong , B.C. , Nestor , A. : How are visual words represented? insights from eeg-based visual word decoding, feature derivation and image reconstruction . Human brain mapping 40 ( 17 ), 5056 - 5068 ( 2019 )

10. Linzen , T. : How can we accelerate progress towards human-like linguistic generalization? arXiv preprint arXiv: 2005 . 00955 ( 2020 )

11. Murphy , B. , Wehbe , L. , Fyshe , A. : Decoding language from the brain . Language, cognition, and computational models pp. 53 - 80 ( 2018 )

12. Nunez , P.L. : Brain, mind, and the structure of reality . Oxford University Press ( 2012 )

13. Oliveira Filho , F. , Cruz , J.L. , Zebende , G. : Analysis of the eeg bio-signals during the reading task by dfa method . Physica A: Statistical Mechanics and its Applications 525 , 664 - 671 ( 2019 )

14. Oseki , Y. , Asahara , M. : Design of bccwj-eeg: Balanced corpus with human electroencephalography . In: Proceedings of the 12th Language Resources and Evaluation Conference . pp. 189 - 194 ( 2020 )

15. Peng , C.K. , Buldyrev , S.V. , Havlin , S. , Simons , M. , Stanley , H.E. , Goldberger , A.L. : Mosaic organization of dna nucleotides . Physical review e 49(2) , 1685 ( 1994 )

16. Rashid , M. , Sulaiman , N. , PP Abdul Majeed , A. , Musa , R.M. , Bari , B.S. , Khatun , S. , et al.: Current status, challenges, and possible solutions of eeg-based braincomputer interface: a comprehensive review . Frontiers in neurorobotics 14 , 25 ( 2020 )

17. Scholr¨, D. , Zehe , A. , Kobs , K. , Veseli , B. , Westermeier , F. , Bur¨bach, L., Roth , D. , Latoschik , M.E. , Hotho , A. : Improving sentiment analysis with biofeedback data . In: Proceedings of LREC2020 Workshop ” People in language, vision and the mind”(ONION2020) . pp. 28 - 33 ( 2020 )

18. Socher , R. , Perelygin , A. , Wu , J. , Chuang , J. , Manning , C.D. , Ng , A.Y. , Potts , C. : Recursive deep models for semantic compositionality over a sentiment treebank . In: Proceedings of the 2013 conference on empirical methods in natural language processing . pp. 1631 - 1642 ( 2013 )

19. Sun , P. , Anumanchipalli , G.K. , Chang , E.F. : Brain2char: a deep architecture for decoding text from brain recordings . Journal of Neural Engineering 17 ( 6 ), 066015 ( 2020 )

20. Zebende , G.F. , Oliveira Filho , F.M. , Leyva Cruz , J.A. : Auto-correlation in the motor/imaginary human eeg signals: A vision about the fdfa fluctuations . PloS one 12(9) , e0183121 ( 2017 )

21. Zhong , Q. , Zhu , Y. , Cai , D. , Xiao , L. , Zhang , H.: Electroencephalogram access for emotion recognition based on a deep hybrid network . Frontiers in Human Neuroscience 14 ( 2020 )

22. Zorick , T. , Mandelkern , M.A. : Multifractal detrended fluctuation analysis of human eeg: preliminary investigation and comparison with the wavelet transform modulus maxima technique . PloS one 8 ( 7 ), e68360 ( 2013 )