Streaming Gait Assessment for Parkinson’s Disease Cristopher Flagg Ophir Frieder Georgetown University Georgetown University cris@ir.cs.georgetown.edu ophir@ir.cs.georgetown.edu Sean MacAvaney Gholam Motamedi Georgetown University Georgetown MedStar Hospital sean@ir.cs.georgetown.edu motamedi@gunet.georgetown.edu Figure 1: Analysis of the normality of the gait of a patient with Parkinson’s disease. ABSTRACT feet with relatively small surface area requires very complex and Patients with progressive neurological disorders such as Parkin- delicate interactions between the musculoskeletal system on the son’s disease, Huntington’s disease, and Amyotrophic Lateral Scle- one hand, the peripheral nervous system (PNS) and the central rosis (ALS) suffer both chronic and episodic difficulties with locomo- nervous system (CNS) on the other. This task becomes even more tion. These difficulties result in falls and injuries which negatively complicated considering that while walking the whole body stands affect a patient’s quality of life. Decision support within the health only on one foot while the other foot is lifted and has to be put domain attempts to characterize the patient’s current gait with re- back on the ground in synchrony with the other foot. spect to recent and long term gait characteristics to monitor disease Patients with Parkinson’s disease or Parkinsonism (a set of simi- degeneration and suggest preventative intervention. We propose lar neurodegenerative disorders) are increasingly at risk of falling the application of an attention based bi-directional recurrent neu- for a variety of mechanisms involved. By freezing of the upper ral network (RNN) to medical gait data collected from wearable body during a walk, they would be thrown forward and given their mobile sensors to identify and rate the normality of gait patterns slowed postural reflexes they would fall; they develop rapid, small, from streaming data and to inform clinicians of specific gait abnor- shuffling steps and a tendency to run (festination). As the disease malities. Experimental results with respect to multiple data sets progresses, movements are further impaired leading to stiffness demonstrate the effectiveness of streaming gait analysis to augment and episodic immobility known as “freezing of gait" (FOG) [16]. traditional health care diagnostic methods, automatically classify a A recent multi-study, multi-regional estimate for individuals over patient’s mobility, and provide monitoring of patients outside of the age of 45 suggests there are 572 individuals with Parkinson’s the clinical environment. disease per 100,000 people [13] or over 1.5 million individuals with Parkinson’s disease in the United States today. CCS CONCEPTS It is important to provide a continuous assessment of the quality of the patient’s gait to assign both an instant assessment of the • Applied computing → Health care information systems; normality of a patient’s gait in a clinical setting and an historical Health informatics; • Computing methodologies → Causal context by which changes in the gait may be assessed over time. reasoning and diagnostics; Neural networks. The medical group associated with this study, MedStar Hospital KEYWORDS and the Georgetown University Department of Neurology, supports over 6500 Parkinson’s patients and provides care for all manner of Neural Networks, Parkinson’s Disease, Gait, Diagnostics neurological conditions that involve gait stability, including stroke service, neuromuscular disorders, and movement disorders which affect gait such as Parkinson’s, dystonia, tremors, and cerebellar 1 INTRODUCTION abnormalities. In conjunction with the diagnosis and management of such conditions using traditional methods, we developed a neural Human gait is controlled by a complex set of interactions between network for automated gait analysis that provides a diagnostic tool multiple organ systems. Keeping balance while standing on two for pre-clinical assessment and improved diagnosis. As a means Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons of continuous monitoring, our neural network for automated gait License Attribution 4.0 International (CC BY 4.0). Flagg, Frieder, MacAvaney, and Motamedi analysis provides historical insight and long term monitoring for Attention based RNNs are also used to study video gait silhouettes decision support and preventative intervention. [11]. Silhouettes are generated from a video sequence, and view- Figure 1 shows an example of the analysis of a gait for a Parkin- independent features are then generated for the gait. In this context, son’s disease patient. The gray box indicates the one second window gait irregularities refer to individually identifiable gait features of data used to estimate the gait normality of the window, where the within a video sequence but not specific degeneration of any type. normality is indicated by the point located in the upper right corner. Finally, EEG signals used for authentication [19] were employed The cross-over from right to left foot shows a strong similarity to based on a 1D Convolutional Long Short-Term Memory Neural the cross-over of a normal gait. Network (1D-Convolutional LSTM). The network decodes the EEG Any deterioration in the structures involved in this process using four levels of 1D convolutions prior to feeding the resultant may result in gait abnormality. As a result, gait abnormalities may vectors into the LSTM. present in many different ways. For example, peripheral neuropa- Analysis of activity context and activity recognition spans the thy as a very common condition (affecting 2.4% of the population, gamut of neural network implementations [2, 17]. The analysis but by age rising up to 8%) can interfere with gait stability by in- is usually derived from accelerometer and gyroscope data from terfering with signal transduction (in particular deep positional cell phones worn on the subject at a specific location. These data sensation carried by thick myelinated nerves) to the CNS. Patients sets [1] typically include walking (on both horizontal and inclined with severe peripheral neuropathy may not feel their position in surfaces), descending and ascending stairs, jumping, running or space fast enough to correct their position and fall during walking. jogging, sitting and standing. Activities of daily living include run- Previous attempts to automatically distinguish degenerate gait ning or ascending stairs. Hammerela[7] explores deep, convolu- from normal gait are hindered by two factors. First, previous re- tional, and recurrent approaches across three representative data search treats degenerate gait as always degenerate and normal gait sets that contain movement data captured with wearable sensors as always normal. Degeneration of gait is a gradual progression, to different tasks. These data sets only focus on asymptomatic or resulting in only a portion of the gait suffering from abnormalities. Non-Parkinsonian subjects. Events such as FOG are episodic and occur at random intervals. Attention-based gait recognition is approached as a WiFi re- The progression gait degeneration provides indicators that may not flectance problem [23] where spectrograms are generated from the be apparent from a single clinical session with a subject. signal reflection from eleven walking subjects. Multiple WiFi ac- Second, publicly available data sets focusing on Parkinson’s cess points determine signals and feed parameters into bi-direction disease gait provide both raw sensor data as well as information RNN with attention. The research describes an encoder-decoder derived from this raw sensor data. These data include analysis format for the network, but then claims the decoder has no input, and derived signals that are simply not available without further reducing the network to an RNN with attention. Other systems offline processing of the raw data, which is not appropriate for [25] analyze gaits and brainwaves of seven participants using an a streaming or online context. Some efforts automatically extract encoder-decoder network with attention. The decoder does not these parameters in real-time [14] for the purpose of monitoring generate a sequence of output, so it is unclear what data is fed into and analysis, and while an improvement over offline analysis, is the final fully connected network, other than the attention vector. still only an abstraction of the raw sensor data. Given the goals of automatically classifying a patient’s mobility and monitoring patients outside of the clinical environment we 2.2 Clinical Gait Analysis validate the following hypotheses: Jovanov[10] discloses a wearable system for real-time gait moni- H1: Streaming gait analysis can rate the level or normality within toring to recognize FOG episodes. They recorded signals from five an individual’s gait pattern. experiments, four from simulated freezing gait events, and one from H2: Streaming gait analysis can identify specific portions of an the real patient and analyzed feasibility of the real-time detection. individual’s gait pattern that suffer from degeneration. Joshi[9] presents the automatic noninvasive identification of H3: Streaming gait analysis can categorize degradation in an indi- Parkinson’s disease based on spatio temporal gait variables. The vidual’s gait pattern over time. authors use wavelet transform and a support vector machine (SVM) to produce efficient classification based on a representation of spatio temporal gait variables to identify Parkinson’s gait. 2 RELATED WORK Shetty[18] focuses on the specific gait characteristics which would help differentiate Parkinson’s Disease from other neurologi- 2.1 Non-Clinical Gait Analysis using Neural cal diseases (Amyotrophic Lateral Sclerosis (ALS) and Huntington’s Networks disease) as well as healthy controls. A range of statistical feature The re-identification of an individual is typically framed as a video vectors are considered from the time series gait data which are gait analysis problem. Sequential frames of a video are used to clas- then reduced using a correlation matrix. These feature vectors are sify individuals. Recent work utilizes bi-directional RNNs both with then individually analysed to extract the best seven feature vectors [12] and without [22] attention mechanisms. Other re-identification which are then classified using a Gaussian radial basis function techniques focus on gait analysis from multiple sensors placed on kernel based support vector machine (SVM) classifier. the body (foot, thigh, and lower back) [26]. Wu[21] uses stride interval parameters to form a feature vector The authentication is performed using an RNN to process the in the pattern classification experiments. The results evaluated with raw features and a CNN to perform the final authentication [5]. the leave-one-out cross-validation method demonstrated that the Streaming Gait Assessment for Parkinson’s Disease least squares support vector machine with polynomial kernels was 3 DATA SETS able to provide an accurate classification. 3.1 Gait in Neurodegenerative Disease Database (GaitNDD) 2.3 Clinical Gait Analysis using Neural This publicly available data set focuses on the pathophysiology Networks neurodegenerative diseases to improve the "ability to measure re- In research relating to gait analysis, a form of testing referred to as sponses to therapeutic interventions."[8] Control subjects (n = 16) “all-training, all-testing" is utilized. This simply refers to training are considered to have the optimal normal gait while subjects with on all of the data and using the validation set in lieu of the testing Parkinson’s disease (n = 15), Huntington’s disease (n = 20), or Amy- set. For smaller data sets, a testing method called “leave one out" is otrophic Lateral Sclerosis (n = 13) are considered to have abnormal used, where all but one sample is used for training and validation, gaits. The subjects in this data set with Parkinson’s disease were and the held-out sample is used for testing. In research where this professionally assessed and ranked on the Hoehn and Yahr scale. method is used there is no strong definition of what is left out. Subjects were required to walk independently for five min, did not In several related papers, [27] and [28], a bi-directional RNN is require an assistive device for mobility, and were free from other utilized for analysis of gait data. The foot pressure signals are used gait affecting pathologies. The study was approved by the Mas- in conjunction with derived data relating to swing and stance. This sachusetts General Hospital Institutional Review Board and made research compares each of three degenerative conditions to the publicly available through PhysioNet on December 21st, 2000. control gait using both the “all-training, all-testing" and “leave one The data for this data set were obtained through force-sensitive out" testing methodologies. These results are indicative of neural resistors placed under each subject’s foot. Stride-to-stride measures network overfitting and use both raw and derived data not available of footfall contact times were derived from these signals. The data in a streaming context. are divided into subjects in the control group, with Parkinson’s A cross-correlation-based feature extraction and Elman’s re- disease, with ALS, and with Huntington’s disease. current neural network (ERNN) based classification [3] is used to To simulate a streaming data scenario, we only use the force partition healthy and pathological gaits, followed by partitioning of sensor readings. The additional derived signals were not used as pathological gaits into Parkinson’s disease, ALS, and Huntington’s part of the data for analysis. disease. The research uses a 50% training and a 50% testing split and suffers from the issues raised in the “all-training, all-testing" 3.2 Gait in Parkinson’s Disease (GaitPDB) methodology. The research notes a direct visual analysis of foot This publicly available data set contains measures of gait from 93 pressure measurements “reveals that it is impossible to differentiate subjects with idiopathic Parkinson’s disease and 73 control subjects. between healthy and pathological subjects without any ambigu- The database includes the vertical ground reaction force from eight ity." The inability to visualize gait abnormality is a result of the foot pressure sensors recorded for subjects as they walked at a self- methodology used, which is addressed by this paper. selected pace for approximately two minutes on level ground. [6] For predicting FOG experienced by patients with Parkinson’s These data were collected at the Laboratory for Gait Neurodynam- disease [20], an LSTM with a 50% training and a 50% testing split is ics, Movement Disorders Unit of the Tel-Aviv Sourasky Medical used to create an overall FOG identifier. Transfer learning applies a Center and made available on February 25th, 2008. new layer of the network which is trained on a portion of a held-out The data also includes two additional data points that reflect the subject’s data and then tested on another portion of that subject’s sum of the left and right foot pressures. To simulate a streaming data to identify the user. data scenario, we only use the summed force sensor readings to A multi-layered artificial neural network was constructed to match the two force readings provided by the GaitNDD data set. classify control, Parkinson’s disease, ALS, and Huntington’s dis- The additional force sensor readings and the derived signals are ease using “One-versus-one", “one-versus-rest", and “control-versus- not used as part of the data for analysis. Additionally, this data set pathological" analysis. The research includes an overview of the includes 214 total trials with Parkinson’s participants and 92 total results obtained by previous papers using the “Gait in Neurodegen- trials with control participants. erative Disease Database" data set. Other classification utilizes a three layer Radial Basis Function 4 EXPERIMENTAL METHODS (RBF) activation based neural network [24]. The authors feature vector sequences for all the 93 Parkinson’s disease patients and 73 4.1 Gait Sensors and Normalization healthy controls and then extract all of the subject’s gait features The GaitNDD and GaitPDB data sets provide left and right binary as a time series provided as the input of the RBF neural network. foot pressure signals which are appropriate incoming data in a Mohammadian[15] proposes a deep normative modeling as a streaming context. Only the force sensor data from each data set probabilistic novelty detection method, in which a model of the are used. The initial ten seconds of the time series include some distribution of normal human movements is recorded by wearable standing and non-walking measurements and data before this point sensors to detect abnormal movements in patients with Parkinson’s are not used in these experiments. disease and ALS. The problem is framed as a novelty detection Each stream of data, comprising left and right foot pressures, is framework where a movement disorder behavior is treated as an individually normalized within a ten second window to a range extreme of the normal range or, equivalently, as a deviation from from 0 to 1. As the window is passed over the data, a ten second the normal movements. moving average is computed and used to normalize the streaming Flagg, Frieder, MacAvaney, and Motamedi process of generating the final output, which is referred to as the context. In the case of a bi-directional GRU, there are two hidden states for each input: one generated by a forward pass over the window and one generated by the backward pass over the window. We follow the common convention of concatenating the forward and backward results into a single vector. Figure 3: General Attention Applied to Incoming Data Points. The radius of the circles denote the amount of at- tention paid to that point in the gait. Note the increased at- tention at the crossover point where the two foot pressures are equal and the balance shifts from one foot to the other. Figure 2: Bi-Directional GRU with General Attention. data. Any incoming data that exceeded the [0,1] bounds are clipped Once the GRU has processed the incoming data and created a hid- to the maximum or minimum value. These normalized data are den state for each incoming sample, a general attention mechanism then used as the input to the neural network. is applied. A single fully connected layer with an input dimension The model used to identify normal patterns within a subject’s that matches the hidden state size of the GRU and an output of one gait contains three layers: 1) A Gated Recurrent Unit (GRU), 2) A dimension is used as the attention layer. This layer is applied to the general attention layer to summarize the GRU output, and 3) A hidden vectors and creates a single value representing the strength final fully connected layer. or attention of the hidden state that matches the corresponding input. This value is multiplied against the hidden state to increase 4.2 Network Design or decrease the strength of the hidden state with respect to the A Gated Recurrent Unit (GRU) is a variant of the Long Short Term attention. These modified hidden states are summed to create the Memory (LSTM) RNN structure. Internally, the GRU uses an update final output of the GRU with respect to the general attention vector. and reset gate to determine which information should be passed Since the left and right sensor values are combined together by the to output. These gates determine how much information from GRU, the attention is applied at each time step, rather than to a previous data should be saved as well as how the saved data are particular sensor reading, as shown in Figure 3. combined with the incoming data to produce the output. In this The final GRU output with applied attention is then fed into the manner, the output of previous time steps is combined with the final fully connected layer. This layer outputs a value relating the current input. The full model is shown in Figure 2. normality of the data within the window where [1] is completely As the data arrive at the model, one second (30 left/right data normal and [0] is completely degenerate. The ground truth for this points) of samples are grouped into a window and passed to the value is determined by the file from which the data came: data GRU. Preliminary studies have shown a one second window pro- specified as Control are assigned a target value of [1] and data vides optimal results with these gait analysis methods. A sliding other files (ALS, Parkinson’s disease, and Huntington’s disease) are window is used for this analysis, however other windowing meth- assigned a target value of [0]. Although our premise is that gait ods such as discrete windows could also be used. Each data point is not a binary assignment of normal/degenerate across an entire passed to the GRU produces an output which is fed into the next subject’s data set, the use of binary classification labels to train the iteration of the GRU along with the next data point. These outputs model provides enough insight into difference between normal and are generally considered hidden states since they are part of the degenerate cases to train an accurate model. Streaming Gait Assessment for Parkinson’s Disease GaitNDD 4.4 Implementation Control Parkinson’s Huntington’s ALS The PyTorch implementation utilizes an initial three layered bi- Total 16 15 12 13 directional GRU with 256 neurons in each hidden layer. A dropout Train 12 4 4 4 of 0.3 is used by the GRU to reduce overfitting. The input size is Test 4 4 4 4 two, with one channel for the left signal data and one channel for the right signal data. The hidden vectors for each batch contained GaitPDB one hidden layer for the forward RNN and one for the backwards Control Parkinson’s Huntington’s ALS RNN, and are subjected to a general attention vector of size 512. Total 73 93 - - The attention vector, after application to each context vector, was Train 57 57 - - normalized using softmax and the results are summed to form a Test 16 16 - - single feature vector of 512. This is passed to a fully connected Table 1: Parkinson’s Gait Databases, distribution of training, layer with an output size of one. The learning rate is selected as validation, and testing files. 0.0001 and the cross entropy loss function is used for training. The training batch size is selected as 1024 and the models were trained for 20 epochs. 5 EVALUATION Our evaluation of gait abnormalities focuses specifically on the 4.3 Training/Validation/Testing labeling of individual points within a gait based on the ‘normality’ As shown in Table 1, for all data sets used, a full training, validation, of that point. The model is designed to distinguish between normal and testing split is implemented as follows: and degenerate points within the data. This classification is applied For the GaitNDD data set, an equal number of control (12 files) at three distinct levels of focus. and other files (4 Parkinson’s disease, 4 ALS, and 4 Huntington’s First, an entire gait sequence is reviewed and the overall gait disease) are selected at random for the training set to provide a pattern is classified as either normal or abnormal (one of Parkin- balance of control and abnormal training input. In order to balance son’s disease, ALS, or Huntington’s disease in the case of the Gait- the normal and abnormal cases, the data selection is constrained NDD data set) or the gait is classified as either normal or abnormal by the number of normal gaits within the data set. Using an 80/20 (Parkinson’s disease in the case of the GaitPDB data set). This may split, the first 80% of the data for all of the selected training files be applied by practitioners in a clinical setting to aid in the diagnosis are used as the training set and the final 20% of the data are used of neuromuscular disease. for the validation set. Since gait patterns are repetitive, this creates Additionally, specific portions of the gait sequence may be ana- over-fitting in the validation set. Dropout in the GRU is used to lyzed to identify traits specific to an individual’s gait. This involves lessen the impact of the over-fitting. Due to the limited amount of review of a portion of the gait to identify cyclic irregularities such data available within the data sets, it was not feasible to use disjoint as repeated pressure abnormalities in an otherwise normal gait or sets of subjects for the training and validation. portions of a degenerate gait that are not as strongly effected by For the testing set, four files from each category are selected at the neuromuscular disease. random from the files not included in the training set. From these Finally, tracking the normality of a gait over time allows degen- files, the final 20% of the data for all of the selected testing files are eration to be identified. Long term changes in gait are reflected in used for testing to match the data split used for the validation set, the normality as viewed on an hourly, daily, and weekly time frame. with the understanding that the entire testing set could be used for Since the data sets do not include long-term tracking of gait, the testing. Due to the repetition of the gait within the sessions, the use degeneration of gait over time is simulated by combining different of the final 20% of the testing data did not alter the results when gaits and noting the change in the average normality of the gait. compared to the use of the entire testing set. For the GaitPDB data set, an equal number of control (57 files) 5.1 Identifying abnormal gait and idiopathic Parkinson’s disease (57 files) are selected at random The model predicts the normality of a specific point within a gait to provide a balance of control and abnormal training input. Using based on a sliding window. Within a streaming context, new data an 80/20 split, the first 80% of all of the selected training files is points are received and the sliding window of data points is updated. used as the training set and the remaining 20% of the data is used The updated window is used to predict the normality of the gait at for the validation set. Dropout, again used in the GRU, is used to the new data point. The resulting stream of normality predictions lessen the impact of the over-fitting. is collected to evaluate the overall gait of the subject for monitoring For the testing set, 16 patients with idiopathic Parkinson’s disease and diagnosis. and 16 control patients are selected at random from the files not In the case of the GaitNDD data set, the first 10 seconds of data included in the training set. From these files, the final 20% of the data included non-gait-events such as standing (high pressure on both are used for testing to match the data split used for the validation feet) and general shifting of balance. The data used for analysis of set, with the understanding that the entire testing set could be used overall gait quality begins after this initial 10 second window. for testing. The use of the final 20% of the testing data did not alter To evaluate the quality of the model when applied to this data the results when compared to the use of the entire testing set. set, the normality ratings of each data point within the session is Flagg, Frieder, MacAvaney, and Motamedi (a) GaitNDD Control Gait Control10, Testing Set (b) GaitNDD Parkinson’s Gait Park11, Testing Set (c) GaitPDB Control Gait GaCo15, Testing Set (d) GaitNDD Parkinson’s Gait JuPt11, Testing Set Figure 4: Normality of Gait Patterns. The blue and green lines indicate the foot pressure exerted by the left and right foot, respectively. The red dots show the confidence of the normality of each sampled window within the gait. collected and the average normality over the gait session is calcu- a more normal prediction (HY assessment of stage one was given a lated. A threshold is established to determine the classification of predicted value of 0.8) and a higher score was given a more abnor- this number. As optimal normal prediction is [1] and the optimal mal prediction (HY assessment of stage four was given a predicted degenerate prediction is [0], the threshold is determined by iden- value of 0.2). Using this gradation of predicted values for subjects tifying the threshold that provides the highest overall precision with Parkinson’s disease drove the average control patient’s predic- over the validation set, in this case 0.45. This average normality is tions toward the threshold since lower stage Parkinson’s patients then assigned to a grouping of normal (normality ≥ threshold) or experience only minimal degradation. degenerate (normality < threshold). The mean squared error (MSE) is included as an additional mea- Gait abnormalities are distributed across the entire session; thus, sure of classification performance. It is the average difference of initial attempts to use a non-binary classification for Parkinson’s the correct (Target) classification value and the value predicted patients resulted in a lower precision and created difficult in iden- (Pred) by the model. The MSE is calculated over all predictions tifying normal subjects. In the initial test, assessed subjects were for the gait session and provides a means of comparing not only assigned predicted normality values based on the inverse of the the specific subject’s classification accuracy but also the accuracy Hoehn and Yahr scale where a patient with a low score was given of the predictions over an entire class of subjects. This method is Streaming Gait Assessment for Parkinson’s Disease particularly useful where the predicted value for a subject lies on a and given a ground truth of [0]. Again, a threshold is determined scale between normal [1] and abnormal [0]. A macro-average of based on the maximum precision obtained over the validation set the MSE for the training and testing sets is provided for each class. to distinguish normal and degenerate sessions. n Table 2 shows the results of the final classification of the Gait- 1Õ MSE = (T arдeti − Predi ) PDB data set. For this data set, any control subjects are considered n i=1 correctly classified if the average normality is above the defined Table 2 shows the results of the final classification of the Gait- threshold and Parkinson’s disease subjects were considered cor- NDD data set. For this data set, any control subjects are considered rectly classified if their average normality is below the threshold. correctly classified if the average normality is above the defined The table shows the number of correct classifications for both the threshold, and all other subjects were considered correctly classified validation and the testing set. The larger data set allows for a higher if their average normality was below the threshold. The table shows testing set prediction accuracy. The MSE is calculated using the the number of correct classifications for both the validation and the same method as the GaitNDD data set. testing set. As seen with previous papers that use the “all testing, all training" methodology, correct classification of all subjects within Subject Normal Abnormal Total Abnormal/All the validation set is easily achieved with this model. The testing control10 14466 2853 17319 0.1648 set shows a high accuracy for classifying the degenerate cases but control16 10837 6482 17319 0.3743 is only able to correctly classify half of the control cases. control2 8440 8879 17319 0.5127 control7 11089 6230 17319 0.3598 GaitNDD GaitPBD Average 44832 24444 69276 0.3528 Class Testing MSE Testing MSE park15 1974 15345 17319 0.8860 Control 2/4 0.1795 16/18 0.1169 park3 8533 8786 17319 0.5073 Parkinson’s 4/4 0.1655 16/18 0.1009 park5 7166 10153 17319 0.5862 Huntington’s 4/4 0.0857 - - park6 820 16499 17319 0.9526 ALS 3/4 0.1230 - - Average 18493 50783 69276 0.7331 All Files 13/16 0.1384 32/36 0.1089 hunt10 3300 14019 17319 0.8095 Table 2: Analysis of the GainNDD and GaitPDB data sets. hunt14 10943 6376 17319 0.3682 The testing columns indicate the number of subjects cor- hunt2 7992 9327 17319 0.5385 rectly classified. The mean squared error (MSE) between the hunt8 11392 5927 17319 0.3422 value predicted by the model and the target value. Average 33627 35649 69276 0.5146 als2 4883 12436 17319 0.7180 als4 5105 12214 17319 0.7052 als5 2405 14914 17319 0.8611 als7 5357 11962 17319 0.6907 Average 17750 51526 69276 0.7438 Table 3: GaitNDD Testing Set. Similar to the supervised ground truth novelty estimate utilized by Mohammadian[15], the percentage of abnormal estimates for the GaitNDD testing set is shown in Table 3. This details the trend observed in individual gaits, where the normal gaits include a set of abnormal points and abnormal gaits include a set of normal points. This distribution of points, shown in Figure 5, illustrates the distribution of estimates across classes. A majority of points within the control set are above the 0.45 threshold and a majority of the Parkinson’s, Huntington’s, and ALS estimates are below the 0.45 threshold. That being said, the points within one standard deviation Figure 5: Distribution of normality estimates by gait type. of the classifications show significant overlap. As a diagnostic tool, Since all gaits are a combination of normal and abnormal the goal is to provide both an overall classification and to identify estimates, this box plot shows the distribution of estimated specific points within the gait that indicate abnormality. normality for each class within the GaitNDD data set. 5.2 Identification of specific gait abnormalities To evaluate the quality of the model when applied to the GaitPDB In addition to the overall classification of gait behavior, assessing data set, the data is divided into two classes: control and Parkinson’s the normality of specific portions of a subject’s gait provides deeper disease. Control sessions are considered normal and given a ground insight into the subject’s overall gait characteristics. This infor- truth of [1]. Parkinsons’s disease sessions are considered degenerate mation may be used for diagnosis and in a clinical review of the Flagg, Frieder, MacAvaney, and Motamedi Figure 6: Simulated Gait Degradation. The red line shows the moving average of the gait normality. The black lines show the average gait normality of the first and second sections with a discontinuity at the point of simulated gait degradation. gait reveals repeating patterns within the gait as well as episodic (starting after 10 seconds) to the splice point in the beginning abnormalities, such as FOG. sequence is added to the new time series. The data from the ending In Figure 4(a), the control gait chosen from the GaitNDD testing sequence from the ending splice point onward are concatenated set exhibits a recurrent degradation after the transfer to the left to the new time series. The point at which the two time series are foot just after the point where the right foot (blue) lifts (reduces joined is recorded as the splice point. pressure) and the left foot (green) steps down (increases pressure) To identify the transition point, there are four cases: a control just before the 118 second mark. This is repeated again at the 102 to control splice, a degenerate to degenerate splice, a control to second, the 128 second, and the 148 second marks. This signal within degenerate splice, and a degenerate to control splice. We generate this subject’s gait is regular and repeating (not episodic). Repeated a random set of 40 transition files, 10 from each of the above cases. abnormalities on a single side of a transfer may not indicate an The average normality for the beginning sequence is used as the abnormality beyond an injury. Other possible causes could include threshold for identifying changes to the gait. A change in long term imbalances such as a strong and weak side related to handedness. gait is defined as a change of average gait normality (over the one In Figure 4(b), the GaitNDD Parkinson’s testing gait seems to second window) of 25%. have shuffling gait issues and maybe even some festination (a ten- As shown in Figure 6, changes in normality are identified around dency to speed up in parallel with a loss of normal amplitude of the splice point. In the cases where the average normality of the repetitive movement, e.g. marche a petits pas). The transfer from beginning sequence is within 20% of the average normality of the the right foot (blue) to the left foot (green) exhibits some hesitation ending sequence, the system is able to identify the gait degradation. characterized by a jagged transfer. This shows that there is hesi- When the difference in average normality is less than 20% the tance in the transfer from the left foot to the right foot. While the system is not able to consistently identify the degradation. While session shows low normality over a majority of the gait, a highly not a long term analysis, it does demonstrate the ability to evaluate regular gait interval is indicated by the gray window which covers changes to a patient’s gait. the transfer from the left foot (green) to the right foot (blue). In Figure 4(c), a normal gait from the GaitPDB data set shows 6 CONCLUSION a stumble or inconsistency between 18 and 21 seconds. This is an Automated gait analysis provides decision support and aids in the episodic variation that is not exhibited in the surrounding steps. The diagnosis of neurodegenerative diseases. The pre-clinical and clin- system identifies this as a portion of the gait with low normality. ical assessment of the overall normality of a subject’s gait using in Figure 4(d), no window is used since the abnormality from sensor data from a wearable device can improve the initial diagnosis seven seconds to tweleve seconds is clearly identified by the system. of Parkinsonian gait. We demonstrate this assessment using a recur- While there is a generally low abnormality across the data set, this rent neural network architecture with attention. While this style particularly low normality could indicate a stumble or hesitance in of network is known, the application of normality analysis could transition from the left foot (green) to the right foot (blue). have a considerable impact on a subject’s prognosis and improve their overall quality of life. 5.3 Simulation of Degradation Streaming data from a wearable device makes it possible to mon- The publicly available data sets limit the scope of the data to a itor disease degeneration and suggest preventative intervention single subject during a single clinical session and do not include over an extended period of time. This data may also act as an in- long-term degradation data. Long term activities of daily living data dicator that more serious clinical review is in order. Moving to sets are limited to only normal candidates. To simulate long-term an approach that rates the normality of the gait gives doctors the degradation, the testing files from the GaitNDD database are spliced flexibility to review a subject’s gait in a clinical setting, identify to create a transition between two different gaits. specific issues within a subject’s gait, as well as provide long term First, two files are chosen at random and are considered the monitoring for continued gait degradation. This enables doctors to beginning sequence and the ending sequence. A single point is increase the quality of care they provide to patients with neurogen- chosen at random within each of the files. The nearest transition erative diseases and provides a continuous monitoring paradigm from left foot to right foot, where the pressure for each foot is for patients. approximately equal, is then selected in each file as the point at which to splice the files. The data from the beginning sequence Streaming Gait Assessment for Parkinson’s Disease REFERENCES [21] Yunfeng Wu and Sridhar Krishnan. 2009. Statistical analysis of gait rhythm [1] Marc Bächlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jeffrey M Hausdorff, in patients with Parkinson’s disease. IEEE Transactions on Neural Systems and Nir Giladi, and Gerhard Tröster. 2010. Wearable assistant for Parkinson’s disease Rehabilitation Engineering 18, 2 (2009), 150–158. patients with the freezing of gait symptom. IEEE Trans. Information Technology [22] Yang Wu, Jie Qiu, Jun Takamatsu, and Tsukasa Ogasawara. 2018. Temporal- in Biomedicine 14, 2 (2010), 436–446. enhanced convolutional network for person re-identification. In Thirty-Second [2] Zartasha Baloch, Faisal Karim Shaikh, and Mukhtiar Ali Unar. 2019. DEEP AAAI Conference on Artificial Intelligence. ARCHITECTURES FOR HUMAN ACTIVITY RECOGNITION USING SENSORS. [23] Yang Xu, Min Chen, Wei Yang, Sheng Chen, and Liusheng Huang. 2018. Attention- 3C Tecnologia (2019). based Walking Gait and Direction Recognition in Wi-Fi Networks. arXiv preprint [3] Saibal Dutta, Amitava Chatterjee, and Sugata Munshi. 2009. An automated arXiv:1811.07162 (2018). hierarchical gait pattern identification tool employing cross-correlation-based [24] Wei Zeng, Chengzhi Yuan, Qinghui Wang, Fenglin Liu, and Ying Wang. 2019. feature extraction and recurrent neural network based classification. Expert Classification of gait patterns between patients with Parkinson’s disease and systems 26, 2 (2009), 202–217. healthy controls using phase space reconstruction (PSR), empirical mode decom- [4] Carsten Eickhoff, Yubin Kim, and Ryen White. 2020. Overview of the Health position (EMD) and neural networks. Neural Networks 111 (2019), 64–76. Search and Data Mining (HSDM 2020) Workshop. In Proceedings of the Thirteenth [25] Xiang Zhang, Lina Yao, Chaoran Huang, Tao Gu, Zheng Yang, and Yunhao Liu. ACM International Conference on Web Search and Data Mining (WSDM ’20). ACM, 2017. DeepKey: An EEG and gait based dual-authentication system. arXiv preprint New York, NY, USA. https://doi.org/10.1145/3336191.3371879 arXiv:1706.01606 (2017). [5] Giacomo Giorgi, Fabio Martinelli, Andrea Saracino, and Mina Sheikhalishahi. [26] Yuting Zhang, Gang Pan, Kui Jia, Minlong Lu, Yueming Wang, and Zhaohui Wu. 2018. Walking Through the Deep: Gait Analysis for User Authentication Through 2014. Accelerometer-based gait recognition by sparse representation of signature Deep Learning. In IFIP International Conference on ICT Systems Security and points with clusters. IEEE transactions on cybernetics 45, 9 (2014), 1864–1875. Privacy Protection. Springer, 62–76. [27] Aite Zhao, Lin Qi, Junyu Dong, and Hui Yu. 2018. Dual channel LSTM based [6] Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. multi-feature extraction in gait for diagnosis of Neurodegenerative diseases. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, Knowledge-Based Systems 145 (2018), 91–97. and H. Eugene Stanley. 2000. PhysioBank, PhysioToolkit, and PhysioNet. Cir- [28] Aite Zhao, Lin Qi, Jie Li, Junyu Dong, and Hui Yu. 2018. LSTM for diagnosis of culation 101, 23 (2000), e215–e220. https://doi.org/10.1161/01.CIR.101.23.e215 neurodegenerative diseases using gait data. In Ninth International Conference on arXiv:https://www.ahajournals.org/doi/pdf/10.1161/01.CIR.101.23.e215 Graphic and Image Processing (ICGIP 2017), Vol. 10615. International Society for [7] Nils Y Hammerla, Shane Halloran, and Thomas Plötz. 2016. Deep, convolutional, Optics and Photonics, 106155B. and recurrent models for human activity recognition using wearables. arXiv preprint arXiv:1604.08880 (2016). [8] Jeffrey M. Hausdorff, Apinya Lertratanakul, Merit E. Cudkowicz, Amie L. Pe- terson, David Kaliton, and Ary L. Goldberger. 2000. Dynamic markers of al- tered gait rhythm in amyotrophic lateral sclerosis. Journal of Applied Phys- iology 88, 6 (2000), 2045–2053. https://doi.org/10.1152/jappl.2000.88.6.2045 arXiv:https://doi.org/10.1152/jappl.2000.88.6.2045 PMID: 10846017. [9] Deepak Joshi, Aayushi Khajuria, and Pradeep Joshi. 2017. An automatic non- invasive method for Parkinson’s disease classification. Computer methods and programs in biomedicine 145 (2017), 135–145. [10] E Jovanov, E Wang, L Verhagen, M Fredrickson, and R Fratangelo. 2009. deFOG—A real time system for detection and unfreezing of gait of Parkinson’s patients. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 5151–5154. [11] Shuangqun Li, Wu Liu, and Huadong Ma. 2019. Attentive Spatial-Temporal Summary Networks for Feature Learning in Irregular Gait Recognition. IEEE Transactions on Multimedia (2019). [12] Zhihui Li, Lina Yao, Feiping Nie, Dingwen Zhang, and Min Xu. 2018. Multi- rate gated recurrent convolutional networks for video-based pedestrian re- identification. In Thirty-Second AAAI Conference on Artificial Intelligence. [13] C Marras, JC Beck, JH Bower, E Roberts, B Ritz, GW Ross, RD Abbott, R Savica, SK Van Den Eeden, AW Willis, et al. 2018. Prevalence of Parkinson’s disease across North America. NPJ Parkinson’s disease 4, 1 (2018), 21. [14] Ilaria Mileti, Marco Germanotta, Enrica Di Sipio, Isabella Imbimbo, Alessan- dra Pacilli, Carmen Erra, Martina Petracca, Stefano Rossi, Zaccaria Del Prete, Anna Rita Bentivoglio, Luca Padua, and Eduardo Palermo. 2018. Measuring Gait Quality in Parkinson’s Disease through Real-Time Gait Phase Recognition. Sensors (Basel, Switzerland) 18, 3 (20 Mar 2018), 919. https://doi.org/10.3390/ s18030919 29558410[pmid]. [15] Nastaran Mohammadian Rad, Twan van Laarhoven, Cesare Furlanello, and Elena Marchiori. 2018. Novelty Detection using Deep Normative Modeling for IMU- Based Abnormal Movement Monitoring in Parkinson’s Disease and Autism Spectrum Disorders. Sensors 18, 10 (2018), 3533. [16] John G Nutt, Bastiaan R Bloem, Nir Giladi, Mark Hallett, Fay B Horak, and Alice Nieuwboer. 2011. Freezing of gait: moving forward on a mysterious clinical phenomenon. The Lancet Neurology 10, 8 (2011), 734–744. [17] Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D Lane, Cecilia Mascolo, Mahesh K Marina, and Fahim Kawsar. 2018. Multimodal deep learning for activity and context recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 157. [18] Sachin Shetty and YS Rao. 2016. SVM based machine learning approach to identify Parkinson’s disease using gait analysis. In 2016 International Conference on Inventive Computation Technologies (ICICT), Vol. 2. IEEE, 1–5. [19] Yingnan Sun, Frank P-W Lo, and Benny Lo. 2019. EEG-based user identification system using 1D-convolutional long short-term memory neural networks. Expert Systems with Applications 125 (2019), 259–267. [20] Vishwas G Torvi, Aditya Bhattacharya, and Shayok Chakraborty. 2018. Deep Domain Adaptation to Predict Freezing of Gait in Patients with Parkinson’s Disease. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 1001–1006.