Amyotrophic lateral sclerosis (ALS) is a progressive neurodegenerative disorder that results in the gradual deterioration of motor abilities, leading to challenges in breathing, speaking, swallowing, and ultimately death, typically occurring within a few years. The symptoms of ALS can vary significantly from one individual to another, afecting various bodily functions and areas. To assess this wide range of symptoms, the Amyotrophic Lateral Sclerosis Functional Rating Scale - Revised (ALSFRS-R) is utilized. Predicting the ALSFRS-R score is clinically relevant for personalizing patient monitoring. To address this need, the Intelligent Disease Progression Prediction challenge was organized, tasking participants with developing novel methods to predict these scores using non-invasive sensor data that monitor some individual characteristics. The competition included two tasks that difered only in the way the ALSFRS-R questionnaires were completed: either by medical staf (task 1) or by the patient (task 2). Given the limited number of patients on the training set, it was decided to use a relatively simple model, Random Forest, and to preselect sensor features by retaining those most correlated with the outcome to be predicted. We selected the model with the lowest MAE estimated by cross-validation on the challenge training set. The competition results demonstrate that our method attained on the test set an average Mean Absolute Error (MAE) of 0.234 and 0.311, along with a Root Mean Square Error (RMSE) of 0.519 and 0.601 for tasks 1 and 2, respectively. Although the error may appear very low, this is because questionnaire values tend to remain constant from one visit to another, thus facilitating prediction.
of this paper is to utilize information collected through the sensors of a commercial fitness smartwatch,
past ALSFRS-R scores, and static features (such as age, sex, etc.) to predict future ALSFRS-R scores. The
questionnaire data available has two diferent sources: they can be filled out by a doctor or by the patient
through the use of a dedicated smartphone application. Therefore, the challenge has been divided into
two tasks with the same goal but using data from diferent sources characterized by diferent frequencies
of intervals between one questionnaire and the next, as well as difering medical or personal opinions,
which may lead to diferent scoring choices despite similar symptoms. To solve these tasks, classical
machine learning models were used instead of deep learning given the small number of patients in the
training dataset. For more details we refer the reader to the challenge overview papers [
The paper is divided into the following sections: 2 Related Work, which reports some papers addressing topics similar to this work; 3 Methodology, where the entire procedure that led to the predictions for the two tasks is outlined; 4 Experimental Setup, which details the procedures used; 5 Results, where the obtained results along with performance metrics are presented; and 6 Conclusions and Future Work, which reviews the essential steps of the paper and proposes alternative methodologies that could be useful for improving predictions.
The quest to identify prognostic factors and build predictive models for amyotrophic lateral sclerosis
(ALS) progression has been a longstanding challenge, but one of paramount importance. ALS exhibits
significant variability in its progression and outcomes, posing obstacles to making accurate predictions.
Many methodologies have undergone rigorous testing using data from the PRO-ACT database. While
this repository may not perfectly capture the full spectrum of ALS patients in the population, it stands
as the largest publicly accessible dataset amalgamating ALS clinical trials [
Wearable devices have been efectively used to study individuals with ALS, demonstrating a link
between ALS progression and behavior and function patterns in people with amyotrophic lateral
sclerosis, as measured by digital wearables [9] [10] [11] [12]. These measurements include total activity
volume, active versus sedentary time, and time spent at home. Additionally, wearable devices are
increasingly utilized to investigate physical activity in populations with cardiovascular disease, multiple
sclerosis, arthritis, and other conditions. Although studies have been conducted to predict characteristics
related to the ALSFRS score, such as its score and slope, to date, there are no predictors leveraging data
from smartwatches to predict the ALSFRS score [13] [
Three diferent types of data were used to predict ALSFRS-R scores: sensor data from Garmin VivoActive 4 smartwatches, static feature data, and ALSFRS-R questionnaire data. Regarding sensor data, these consist of 90 diferent features per day and are characterized by a large number of missing values (in many days, no features were recorded, rendering the sensor vector absent for those days) due to both the data collection device and patient behavior. The static data are baseline characteristics recorded at a specific time, they include: sex, diagnostic delay, age at diagnosis, forced vital capacity (FVC), weight, and body mass index (BMI). Finally, the ALSFRS-R data are of the same type as those to be predicted but collected at a previous time.
To leverage these challenging data, two diferent approaches have been explored: the Mono Window approach and the Double Window approach. The Mono Window approach is the simpler of the two: for each prediction, only the sensors recorded within 7 days prior to the questionnaire to be predicted are used (these can be utilized in various ways, such as averaging or taking the median). The second approach involves considering two sensor data windows instead of one: the first window adjacent to the questionnaire to be predicted, and the second adjacent to the previous available questionnaire. The idea behind this second method is to provide the model with more information about the changes Frequencies of Residues Values in the Training ALSFRS-R Data 0 .3 0 .2 0 le .-1 u a V .0 0 0 . 1 0 . 3
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12
Score name in recorded parameters over time. Despite the large amount of sensor data (13.946 feature vectors) and the fact that the second method seems more natural for handling this type of data, it is heavily penalized by the irregularity of the sensors. Indeed, two 7-day windows with at least 3 days of sensor data were not available in 20 out of 54 patients. For this reason, the choice to use the first approach was mandatory. In this process, for each patient, all sensor data outside the temporal window of the 7 days preceding the ALSFRS-R values to be predicted was disregarded. The sensor data within each time window were averaged along the time to obtain one feature vector of length 90 for each window that is less afected by daily variability. Once the feature vectors representing the sensor data were obtained, they were concatenated with the vectors of static features and with the vectors of previous ALSFRS-R data before the questionnaires to be predicted; by doing this the the final feature vectors were obtained.
Regarding the outcomes to predict, these are the values of ALSFRS-R questionnaires after the last ones available for the training. To solve this task, it has been observed that the questionnaire values tend to remain constant between visits (see Figure 1); therefore, it was decided to use the previous time’s questionnaire score as the prediction baseline and to fit the model on the residuals. To obtain the final prediction value, it was suficient to add the predicted residual to the value of the previous questionnaire relative to the one to be predicted1. The Random Forest Classifier was chosen as the model; unlike deep learning models, it does not require large datasets for training, making it suitable for this task. Before being used to fit the model, the data were preprocessed by scaling and performing feature selection, as explained in the section 4.
The training dataset for this challenge comprises data from 52 diferent patients for both tasks. These data can be categorized into three types: static data, ALSFRS-R data, and data from a smartwatch sensor. By analyzing the correlation matrices, two important facts can be observed: as for the ALSFRS-R data, it can be observed that they can be divided into several correlated groups depending on the area afected by the disease (see Figure 2). Meanwhile, regarding the sensor data, there are strongly correlated features, as shown in the figure 3. As shown in the image, the correlated features are grouped into distinct blocks. For the cardiac features, two main blocks can be identified: the first, smaller block pertains to Heart Rate Variability (HRV), while the second block encompasses other cardiac characteristics related to the RR interval. For the respiratory features, two blocks are associated with respiration and blood oxygenation. Lastly, another block of correlated features pertains to the patient’s steps. The grouping of features 1The samples in the training set corresponding to a residual value occurring fewer than 8 times have been discarded; indeed, they are too few to be recognized by the model. (a) Task 1 (b) Task 2 into these blocks is expected, as they represent diferent statistics describing the same physiological processes.
The two tasks difer only in terms of the subject and the frequency with which the questionnaires are filled out. This makes the two tasks slightly diferent primarily for two reasons: the data from the ifrst task should reasonably be more objective, as it is a clinician who fills out the questionnaires rather than the individual patient. Additionally, the data from the second task are compiled through an app and have therefore a higher and more irregular frequency, as depicted in Figure 4.
Regarding data preprocessing, the features were initially scaled using Min-Max Scaler and imputed with the mean or mode depending on their continuous or categorical nature. Concerning the sensor data, an additional process was added: the correlation between each of these features and the questionnaire to be predicted was calculated, and only those features with a correlation above a certain threshold were retained. This approach was chosen due to the low number of samples available to train the model compared to the total number of features. After the data preprocessing, they were used to train a Random Forest Classifier. To determine the optimal hyperparameters of the model 2, along with the correlation threshold utilized to select the sensor features3, the following cross-validation strategy was employed. The training set of the challenge was divided into an inner training set and a validation set (80-20%). Hyperparameter optimization was conducted via cross-validation on the inner training set using a grid search method4, and the chosen hyperparameters for each the tasks are displayed in Table 1 and Table 2.
The challenge was divided into two tasks, Task 1 and Task 2, each involving the construction of a
model to predict the values of ALSFRS-R questionnaires completed by either a clinician or the patient
using a dedicated smartphone application. Despite testing two diferent macro types of methods: Mono
Window and Double Window, the low number of patients in the training set resulted in significantly
lower performance for the second type. Therefore, we only submitted results from the first type. These
2The Random Forest hyperparameters tested were maxing deep and max features; they have been tested respectively in ranges
[
Sensors correlation matrix in the Training set
The models attempted to predict the ALSFRS-R questionnaire values were constrained by the small size of the training dataset. Despite experimenting with models utilizing various time intervals, the only ones proving useful for prediction were those solely relying on a window of sensor data adjacent to the questionnaire to be predicted, without leveraging information from more distant times. This limitation stems from the challenging nature of the data, which contains a large number of missing values. Among the models tested, the one demonstrating the best performance and subsequently used for submission was based on Random Forest, preceded by a feature selection step to reduce the number of sensor features. (a) Task 1 50 100 200 250 300 0 50
The performance obtained shows significant variability depending on the questionnaire number; The Mean Absolute Error (MAE) calculated on the test set is 0.23 and 0.31 respectively for Task 1 and Task 2, while the Root Mean Squared Error (RMSE) is 0.52 and 0.60 respectively. The lower error is observed in the first task, which could be attributed to the fact that the questionnaire compilation by clinical staf tends to be more reliable and objective compared to the subjective opinion from the patient. These seemingly promising results are unfortunately attributed to the ALSFRS-R questionnaires mostly remaining constant from one visit to another, making it very easy to achieve high prediction performance.
To address this issue, one potential approach to improve is using data augmentation to increase the number of questionnaires in the training set. To improve predictions, methods of deep learning could be tested, leveraging much longer sequences of sensor data (such as recurrent neural networks). However, these models require significantly more data for training, which represent the biggest obstacle for this task. MAE (std) learning methods to predict amyotrophic lateral sclerosis disease progression, Scientific reports 12 (2022) 13738. [9] M. Straczkiewicz, M. Karas, S. A. Johnson, K. M. Burke, Z. Scheier, T. B. Royse, N. Calcagno, A. Clark, A. Iyer, J. D. Berry, et al., Upper limb movements as digital biomarkers in people with als, EBioMedicine 101 (2024). [10] L. Garcia-Gancedo, M. L. Kelly, A. Lavrov, J. Parr, R. Hart, R. Marsden, M. R. Turner, K. Talbot, T. Chiwera, C. E. Shaw, et al., Objectively monitoring amyotrophic lateral sclerosis patient symptoms during clinical trials with sensors: observational study, JMIR mHealth and uHealth 7 (2019) e13433. [11] V. Ezeugwu, R. E. Klaren, E. A. Hubbard, P. T. Manns, R. W. Motl, Mobility disability and the pattern of accelerometer-derived sedentary and physical activity behaviors in people with multiple sclerosis, Preventive medicine reports 2 (2015) 241–246. [12] R. P. van Eijk, J. N. Bakers, T. M. Bunte, A. J. de Fockert, M. J. Eijkemans, L. H. van den Berg, Accelerometry for remote monitoring of physical activity in amyotrophic lateral sclerosis: a longitudinal cohort study, Journal of neurology 266 (2019) 2387–2395. [13] A. G. Karanevich, J. M. Statland, B. J. Gajewski, J. He, Using an onset-anchored bayesian hierarchical model to improve predictions for amyotrophic lateral sclerosis disease progression, BMC medical research methodology 18 (2018) 1–13.