Patients with progressive neurological disorders such as Parkinson's disease, Huntington's disease, and Amyotrophic Lateral Sclerosis (ALS) sufer both chronic and episodic dificulties with locomotion. These dificulties result in falls and injuries which negatively afect a patient's quality of life. Decision support within the health domain attempts to characterize the patient's current gait with respect to recent and long term gait characteristics to monitor disease degeneration and suggest preventative intervention. We propose the application of an attention based bi-directional recurrent neural network (RNN) to medical gait data collected from wearable mobile sensors to identify and rate the normality of gait patterns from streaming data and to inform clinicians of specific gait abnormalities. Experimental results with respect to multiple data sets demonstrate the efectiveness of streaming gait analysis to augment traditional health care diagnostic methods, automatically classify a patient's mobility, and provide monitoring of patients outside of the clinical environment.
• Applied computing → Health care information systems;
reasoning and diagnostics; Neural networks.
Human gait is controlled by a complex set of interactions between multiple organ systems. Keeping balance while standing on two feet with relatively small surface area requires very complex and delicate interactions between the musculoskeletal system on the one hand, the peripheral nervous system (PNS) and the central nervous system (CNS) on the other. This task becomes even more complicated considering that while walking the whole body stands only on one foot while the other foot is lifted and has to be put back on the ground in synchrony with the other foot.
Patients with Parkinson’s disease or Parkinsonism (a set of
similar neurodegenerative disorders) are increasingly at risk of falling
for a variety of mechanisms involved. By freezing of the upper
body during a walk, they would be thrown forward and given their
slowed postural reflexes they would fall; they develop rapid, small,
shufling steps and a tendency to run (festination). As the disease
progresses, movements are further impaired leading to stifness
and episodic immobility known as “freezing of gait" (FOG) [
It is important to provide a continuous assessment of the quality of the patient’s gait to assign both an instant assessment of the normality of a patient’s gait in a clinical setting and an historical context by which changes in the gait may be assessed over time. The medical group associated with this study, MedStar Hospital and the Georgetown University Department of Neurology, supports over 6500 Parkinson’s patients and provides care for all manner of neurological conditions that involve gait stability, including stroke service, neuromuscular disorders, and movement disorders which afect gait such as Parkinson’s, dystonia, tremors, and cerebellar abnormalities. In conjunction with the diagnosis and management of such conditions using traditional methods, we developed a neural network for automated gait analysis that provides a diagnostic tool for pre-clinical assessment and improved diagnosis. As a means of continuous monitoring, our neural network for automated gait analysis provides historical insight and long term monitoring for decision support and preventative intervention.
Figure 1 shows an example of the analysis of a gait for a Parkinson’s disease patient. The gray box indicates the one second window of data used to estimate the gait normality of the window, where the normality is indicated by the point located in the upper right corner. The cross-over from right to left foot shows a strong similarity to the cross-over of a normal gait.
Any deterioration in the structures involved in this process may result in gait abnormality. As a result, gait abnormalities may present in many diferent ways. For example, peripheral neuropathy as a very common condition (afecting 2.4% of the population, but by age rising up to 8%) can interfere with gait stability by interfering with signal transduction (in particular deep positional sensation carried by thick myelinated nerves) to the CNS. Patients with severe peripheral neuropathy may not feel their position in space fast enough to correct their position and fall during walking.
Previous attempts to automatically distinguish degenerate gait from normal gait are hindered by two factors. First, previous research treats degenerate gait as always degenerate and normal gait as always normal. Degeneration of gait is a gradual progression, resulting in only a portion of the gait sufering from abnormalities. Events such as FOG are episodic and occur at random intervals. The progression gait degeneration provides indicators that may not be apparent from a single clinical session with a subject.
Second, publicly available data sets focusing on Parkinson’s
disease gait provide both raw sensor data as well as information
derived from this raw sensor data. These data include analysis
and derived signals that are simply not available without further
ofline processing of the raw data, which is not appropriate for
a streaming or online context. Some eforts automatically extract
these parameters in real-time [
Given the goals of automatically classifying a patient’s mobility and monitoring patients outside of the clinical environment we validate the following hypotheses: H1: Streaming gait analysis can rate the level or normality within an individual’s gait pattern.
H2: Streaming gait analysis can identify specific portions of an individual’s gait pattern that sufer from degeneration. H3: Streaming gait analysis can categorize degradation in an individual’s gait pattern over time. 2 2.1
The re-identification of an individual is typically framed as a video
gait analysis problem. Sequential frames of a video are used to
classify individuals. Recent work utilizes bi-directional RNNs both with
[
The authentication is performed using an RNN to process the
raw features and a CNN to perform the final authentication [
Finally, EEG signals used for authentication [
Analysis of activity context and activity recognition spans the
gamut of neural network implementations [
Attention-based gait recognition is approached as a WiFi
relfectance problem [
Jovanov[
Joshi[
Shetty[
Wu[
In research relating to gait analysis, a form of testing referred to as
“all-training, all-testing" is utilized. This simply refers to training
on all of the data and using the validation set in lieu of the testing
set. For smaller data sets, a testing method called “leave one out" is
used, where all but one sample is used for training and validation,
and the held-out sample is used for testing. In research where this
method is used there is no strong definition of what is left out.
In several related papers, [
A cross-correlation-based feature extraction and Elman’s
recurrent neural network (ERNN) based classification [
For predicting FOG experienced by patients with Parkinson’s
disease [
A multi-layered artificial neural network was constructed to classify control, Parkinson’s disease, ALS, and Huntington’s disease using “One-versus-one", “one-versus-rest", and “control-versuspathological" analysis. The research includes an overview of the results obtained by previous papers using the “Gait in Neurodegenerative Disease Database" data set.
Other classification utilizes a three layer Radial Basis Function
(RBF) activation based neural network [
Mohammadian[
This publicly available data set focuses on the pathophysiology
neurodegenerative diseases to improve the "ability to measure
responses to therapeutic interventions."[
The data for this data set were obtained through force-sensitive resistors placed under each subject’s foot. Stride-to-stride measures of footfall contact times were derived from these signals. The data are divided into subjects in the control group, with Parkinson’s disease, with ALS, and with Huntington’s disease.
To simulate a streaming data scenario, we only use the force sensor readings. The additional derived signals were not used as part of the data for analysis. 3.2
This publicly available data set contains measures of gait from 93
subjects with idiopathic Parkinson’s disease and 73 control subjects.
The database includes the vertical ground reaction force from eight
foot pressure sensors recorded for subjects as they walked at a
selfselected pace for approximately two minutes on level ground. [
The data also includes two additional data points that reflect the sum of the left and right foot pressures. To simulate a streaming data scenario, we only use the summed force sensor readings to match the two force readings provided by the GaitNDD data set. The additional force sensor readings and the derived signals are not used as part of the data for analysis. Additionally, this data set includes 214 total trials with Parkinson’s participants and 92 total trials with control participants. 4 4.1
The GaitNDD and GaitPDB data sets provide left and right binary foot pressure signals which are appropriate incoming data in a streaming context. Only the force sensor data from each data set are used. The initial ten seconds of the time series include some standing and non-walking measurements and data before this point are not used in these experiments.
Each stream of data, comprising left and right foot pressures, is
individually normalized within a ten second window to a range
from 0 to 1. As the window is passed over the data, a ten second
moving average is computed and used to normalize the streaming
data. Any incoming data that exceeded the [
The model used to identify normal patterns within a subject’s gait contains three layers: 1) A Gated Recurrent Unit (GRU), 2) A general attention layer to summarize the GRU output, and 3) A ifnal fully connected layer. 4.2
A Gated Recurrent Unit (GRU) is a variant of the Long Short Term Memory (LSTM) RNN structure. Internally, the GRU uses an update and reset gate to determine which information should be passed to output. These gates determine how much information from previous data should be saved as well as how the saved data are combined with the incoming data to produce the output. In this manner, the output of previous time steps is combined with the current input. The full model is shown in Figure 2.
As the data arrive at the model, one second (30 left/right data points) of samples are grouped into a window and passed to the GRU. Preliminary studies have shown a one second window provides optimal results with these gait analysis methods. A sliding window is used for this analysis, however other windowing methods such as discrete windows could also be used. Each data point passed to the GRU produces an output which is fed into the next iteration of the GRU along with the next data point. These outputs are generally considered hidden states since they are part of the process of generating the final output, which is referred to as the context. In the case of a bi-directional GRU, there are two hidden states for each input: one generated by a forward pass over the window and one generated by the backward pass over the window. We follow the common convention of concatenating the forward and backward results into a single vector.
Once the GRU has processed the incoming data and created a hidden state for each incoming sample, a general attention mechanism is applied. A single fully connected layer with an input dimension that matches the hidden state size of the GRU and an output of one dimension is used as the attention layer. This layer is applied to the hidden vectors and creates a single value representing the strength or attention of the hidden state that matches the corresponding input. This value is multiplied against the hidden state to increase or decrease the strength of the hidden state with respect to the attention. These modified hidden states are summed to create the ifnal output of the GRU with respect to the general attention vector. Since the left and right sensor values are combined together by the GRU, the attention is applied at each time step, rather than to a particular sensor reading, as shown in Figure 3.
The final GRU output with applied attention is then fed into the
ifnal fully connected layer. This layer outputs a value relating the
normality of the data within the window where [
As shown in Table 1, for all data sets used, a full training, validation, and testing split is implemented as follows:
For the GaitNDD data set, an equal number of control (12 files) and other files (4 Parkinson’s disease, 4 ALS, and 4 Huntington’s disease) are selected at random for the training set to provide a balance of control and abnormal training input. In order to balance the normal and abnormal cases, the data selection is constrained by the number of normal gaits within the data set. Using an 80/20 split, the first 80% of the data for all of the selected training files are used as the training set and the final 20% of the data are used for the validation set. Since gait patterns are repetitive, this creates over-fitting in the validation set. Dropout in the GRU is used to lessen the impact of the over-fitting. Due to the limited amount of data available within the data sets, it was not feasible to use disjoint sets of subjects for the training and validation.
For the testing set, four files from each category are selected at random from the files not included in the training set. From these ifles, the final 20% of the data for all of the selected testing files are used for testing to match the data split used for the validation set, with the understanding that the entire testing set could be used for testing. Due to the repetition of the gait within the sessions, the use of the final 20% of the testing data did not alter the results when compared to the use of the entire testing set.
For the GaitPDB data set, an equal number of control (57 files) and idiopathic Parkinson’s disease (57 files) are selected at random to provide a balance of control and abnormal training input. Using an 80/20 split, the first 80% of all of the selected training files is used as the training set and the remaining 20% of the data is used for the validation set. Dropout, again used in the GRU, is used to lessen the impact of the over-fitting.
For the testing set, 16 patients with idiopathic Parkinson’s disease and 16 control patients are selected at random from the files not included in the training set. From these files, the final 20% of the data are used for testing to match the data split used for the validation set, with the understanding that the entire testing set could be used for testing. The use of the final 20% of the testing data did not alter the results when compared to the use of the entire testing set. The PyTorch implementation utilizes an initial three layered bidirectional GRU with 256 neurons in each hidden layer. A dropout of 0.3 is used by the GRU to reduce overfitting. The input size is two, with one channel for the left signal data and one channel for the right signal data. The hidden vectors for each batch contained one hidden layer for the forward RNN and one for the backwards RNN, and are subjected to a general attention vector of size 512. The attention vector, after application to each context vector, was normalized using softmax and the results are summed to form a single feature vector of 512. This is passed to a fully connected layer with an output size of one. The learning rate is selected as 0.0001 and the cross entropy loss function is used for training. The training batch size is selected as 1024 and the models were trained for 20 epochs. 5
Our evaluation of gait abnormalities focuses specifically on the labeling of individual points within a gait based on the ‘normality’ of that point. The model is designed to distinguish between normal and degenerate points within the data. This classification is applied at three distinct levels of focus.
First, an entire gait sequence is reviewed and the overall gait pattern is classified as either normal or abnormal (one of Parkinson’s disease, ALS, or Huntington’s disease in the case of the GaitNDD data set) or the gait is classified as either normal or abnormal (Parkinson’s disease in the case of the GaitPDB data set). This may be applied by practitioners in a clinical setting to aid in the diagnosis of neuromuscular disease.
Additionally, specific portions of the gait sequence may be analyzed to identify traits specific to an individual’s gait. This involves review of a portion of the gait to identify cyclic irregularities such as repeated pressure abnormalities in an otherwise normal gait or portions of a degenerate gait that are not as strongly efected by the neuromuscular disease.
Finally, tracking the normality of a gait over time allows degeneration to be identified. Long term changes in gait are reflected in the normality as viewed on an hourly, daily, and weekly time frame. Since the data sets do not include long-term tracking of gait, the degeneration of gait over time is simulated by combining diferent gaits and noting the change in the average normality of the gait. 5.1
The model predicts the normality of a specific point within a gait based on a sliding window. Within a streaming context, new data points are received and the sliding window of data points is updated. The updated window is used to predict the normality of the gait at the new data point. The resulting stream of normality predictions is collected to evaluate the overall gait of the subject for monitoring and diagnosis.
In the case of the GaitNDD data set, the first 10 seconds of data included non-gait-events such as standing (high pressure on both feet) and general shifting of balance. The data used for analysis of overall gait quality begins after this initial 10 second window.
To evaluate the quality of the model when applied to this data set, the normality ratings of each data point within the session is (a) GaitNDD Control Gait Control10, Testing Set (b) GaitNDD Parkinson’s Gait Park11, Testing Set
(c) GaitPDB Control Gait GaCo15, Testing Set
(d) GaitNDD Parkinson’s Gait JuPt11, Testing Set
collected and the average normality over the gait session is
calculated. A threshold is established to determine the classification of
this number. As optimal normal prediction is [
Gait abnormalities are distributed across the entire session; thus, initial attempts to use a non-binary classification for Parkinson’s patients resulted in a lower precision and created dificult in identifying normal subjects. In the initial test, assessed subjects were assigned predicted normality values based on the inverse of the Hoehn and Yahr scale where a patient with a low score was given a more normal prediction (HY assessment of stage one was given a predicted value of 0.8) and a higher score was given a more abnormal prediction (HY assessment of stage four was given a predicted value of 0.2). Using this gradation of predicted values for subjects with Parkinson’s disease drove the average control patient’s predictions toward the threshold since lower stage Parkinson’s patients experience only minimal degradation.
The mean squared error (MSE) is included as an additional
measure of classification performance. It is the average diference of
the correct (Target) classification value and the value predicted
(Pred) by the model. The MSE is calculated over all predictions
for the gait session and provides a means of comparing not only
the specific subject’s classification accuracy but also the accuracy
of the predictions over an entire class of subjects. This method is
particularly useful where the predicted value for a subject lies on a
scale between normal [
1 Õn MSE =
(T arдeti − Predi ) n i=1
Table 2 shows the results of the final classification of the GaitNDD data set. For this data set, any control subjects are considered correctly classified if the average normality is above the defined threshold, and all other subjects were considered correctly classified if their average normality was below the threshold. The table shows the number of correct classifications for both the validation and the testing set. As seen with previous papers that use the “all testing, all training" methodology, correct classification of all subjects within the validation set is easily achieved with this model. The testing set shows a high accuracy for classifying the degenerate cases but is only able to correctly classify half of the control cases.
To evaluate the quality of the model when applied to the GaitPDB
data set, the data is divided into two classes: control and Parkinson’s
disease. Control sessions are considered normal and given a ground
truth of [
Table 2 shows the results of the final classification of the GaitPDB data set. For this data set, any control subjects are considered correctly classified if the average normality is above the defined threshold and Parkinson’s disease subjects were considered correctly classified if their average normality is below the threshold. The table shows the number of correct classifications for both the validation and the testing set. The larger data set allows for a higher testing set prediction accuracy. The MSE is calculated using the same method as the GaitNDD data set.
control10 control16 control2 control7
park15 park3 park5 park6
hunt10 hunt14 hunt2 hunt8
als2 als4 als5 als7
Similar to the supervised ground truth novelty estimate utilized
by Mohammadian[
In addition to the overall classification of gait behavior, assessing the normality of specific portions of a subject’s gait provides deeper insight into the subject’s overall gait characteristics. This information may be used for diagnosis and in a clinical review of the gait reveals repeating patterns within the gait as well as episodic abnormalities, such as FOG.
In Figure 4(a), the control gait chosen from the GaitNDD testing set exhibits a recurrent degradation after the transfer to the left foot just after the point where the right foot (blue) lifts (reduces pressure) and the left foot (green) steps down (increases pressure) just before the 118 second mark. This is repeated again at the 102 second, the 128 second, and the 148 second marks. This signal within this subject’s gait is regular and repeating (not episodic). Repeated abnormalities on a single side of a transfer may not indicate an abnormality beyond an injury. Other possible causes could include imbalances such as a strong and weak side related to handedness.
In Figure 4(b), the GaitNDD Parkinson’s testing gait seems to have shufling gait issues and maybe even some festination (a tendency to speed up in parallel with a loss of normal amplitude of repetitive movement, e.g. marche a petits pas). The transfer from the right foot (blue) to the left foot (green) exhibits some hesitation characterized by a jagged transfer. This shows that there is hesitance in the transfer from the left foot to the right foot. While the session shows low normality over a majority of the gait, a highly regular gait interval is indicated by the gray window which covers the transfer from the left foot (green) to the right foot (blue).
In Figure 4(c), a normal gait from the GaitPDB data set shows a stumble or inconsistency between 18 and 21 seconds. This is an episodic variation that is not exhibited in the surrounding steps. The system identifies this as a portion of the gait with low normality.
in Figure 4(d), no window is used since the abnormality from seven seconds to tweleve seconds is clearly identified by the system. While there is a generally low abnormality across the data set, this particularly low normality could indicate a stumble or hesitance in transition from the left foot (green) to the right foot (blue). 5.3
The publicly available data sets limit the scope of the data to a single subject during a single clinical session and do not include long-term degradation data. Long term activities of daily living data sets are limited to only normal candidates. To simulate long-term degradation, the testing files from the GaitNDD database are spliced to create a transition between two diferent gaits.
First, two files are chosen at random and are considered the beginning sequence and the ending sequence. A single point is chosen at random within each of the files. The nearest transition from left foot to right foot, where the pressure for each foot is approximately equal, is then selected in each file as the point at which to splice the files. The data from the beginning sequence (starting after 10 seconds) to the splice point in the beginning sequence is added to the new time series. The data from the ending sequence from the ending splice point onward are concatenated to the new time series. The point at which the two time series are joined is recorded as the splice point.
To identify the transition point, there are four cases: a control to control splice, a degenerate to degenerate splice, a control to degenerate splice, and a degenerate to control splice. We generate a random set of 40 transition files, 10 from each of the above cases. The average normality for the beginning sequence is used as the threshold for identifying changes to the gait. A change in long term gait is defined as a change of average gait normality (over the one second window) of 25%.
As shown in Figure 6, changes in normality are identified around the splice point. In the cases where the average normality of the beginning sequence is within 20% of the average normality of the ending sequence, the system is able to identify the gait degradation. When the diference in average normality is less than 20% the system is not able to consistently identify the degradation. While not a long term analysis, it does demonstrate the ability to evaluate changes to a patient’s gait. 6
Automated gait analysis provides decision support and aids in the diagnosis of neurodegenerative diseases. The pre-clinical and clinical assessment of the overall normality of a subject’s gait using sensor data from a wearable device can improve the initial diagnosis of Parkinsonian gait. We demonstrate this assessment using a recurrent neural network architecture with attention. While this style of network is known, the application of normality analysis could have a considerable impact on a subject’s prognosis and improve their overall quality of life.
Streaming data from a wearable device makes it possible to monitor disease degeneration and suggest preventative intervention over an extended period of time. This data may also act as an indicator that more serious clinical review is in order. Moving to an approach that rates the normality of the gait gives doctors the lfexibility to review a subject’s gait in a clinical setting, identify specific issues within a subject’s gait, as well as provide long term monitoring for continued gait degradation. This enables doctors to increase the quality of care they provide to patients with neurogenerative diseases and provides a continuous monitoring paradigm for patients.