This technical report details our participation in the iDPP CLEF 2024 challenge, focusing on predictive modeling of ALS progression using sensor data from smartwatches. We competed in Tasks 1 and 2, aiming to predict clinical and self-assessed ALS Functional Rating Scale-Revised (ALSFRS-R) scores. Our methodology centered on machine learning techniques, primarily employing an ensemble of Random Forest classifiers. Our best model, which utilized temporal analysis, achieved a Mean Absolute Error (MAE) of 0.25 in Task 1 and 0.326 in Task 2, with corresponding Root Mean Square Errors (RMSE) of 0.544 and 0.608, respectively. This demonstrates the model's efective leverage of time-series data across diferent assessment settings While the temporal model excelled in capturing the nuances of ALS progression through sensor data, variations in performance between Tasks 1 and 2 highlighted the challenges posed by the subjective nature of self-assessments. Our results contribute to understanding the potential and limitations of current predictive models, emphasizing the importance of sophisticated time-series analysis in improving prognostic tools for ALS.
Amyotrophic Lateral Sclerosis (ALS) is a progressive neurodegenerative disorder that primarily
affects motor neurons in the brain and spinal cord, leading to severe physical disability and ultimately,
respiratory failure [
The ALS Functional Rating Scale-Revised (ALSFRS-R) is a validated instrument used to determine the
degree of functional impairment in ALS patients. It encompasses twelve domains of daily activities, each
rated on a scale from 0 to 4, where higher scores denote greater functional independence. The ALSFRS-R
provides a quantifiable measure of disease progression, correlates significantly with quality of life
measures and is extensively used in clinical trials and practice [
In this context, wearable technology ofers unprecedented opportunities for real-time, objective
monitoring of physiological parameters. Sensor data collected via smartwatches and specialized mobile
applications can provide detailed insights into the physical activity patterns and autonomic functions
of patients, potentially enabling early detection of disease progression markers [
The 2024 Intelligent Disease Progression Prediction (iDPP) challenge at the Conference and Labs
of the Evaluation Forum (CLEF) [
This work describes the participation of our team in iDPP at CLEF 2024, specifically in Tasks 1 and 2.
Advancements in predictive modeling for ALS have utilized machine learning and deep learning to forecast disease understanding and progression efectively.
Seminal work by Hothorn et al. [
On the other hand, recent studies like those by Johnson et al.[
Deep learning has notably advanced ALS research through diverse applications, for example, Van
der Burgh et al. [
The BRAINTEASER project [19] exemplifies the successful integration of clinical assessments with
real-time sensor data, creating comprehensive models that detect nuanced changes in ALS progression,
thereby facilitating more responsive patient care. The iDPP@CLEF challenges in 2023 [20] and the
current challenge [
This study uses data from smartwatches and mobile health applications, focusing on physiological
metrics such as heart rate, steps, and activity levels, alongside patient-reported outcomes. The data
captures continuous physiological data, giving detailed information on many features related to patient
activity and health status over time [
• Corresponding test datasets are used for evaluating model performance.
To ensure the quality and consistency of the input data, a rigorous preprocessing pipeline was implemented. The preprocessing steps included handling missing values, feature engineering, and data cleaning.
Given the longitudinal nature of the data, missing values occurred because some patients did not follow the guidelines every day. To address this issue, we used imputation techniques.
Specifically, Iterative and Simple Imputation were employed to ascribe missing values in both static and sensor datasets [21]. IterativeImputer performs multivariate imputation by modeling each feature with missing values as a function of other features in a round-robin fashion, making it particularly efective for datasets with complex interactions between variables. Additionally, in some cases, mean and median imputation were experimented with for numeric columns, depending on their distribution and specific characteristics. This approach ensures that the imputed values are reasonable and do not introduce significant biases. Finally, rows with more than half of their values missing were excluded to maintain data integrity. This threshold was chosen to balance retaining as much data as possible while ensuring the remaining data was of high quality.
Several approaches were used for feature engineering to enhance the predictive power of the models by creating new features from the raw data to capture underlying patterns and trends more efectively. Particularly, we aggregated sensor data using statistical measures such as mean, median, standard deviation, minimum, maximum, and range. These aggregations were performed over specified time windows to capture temporal dynamics. On the other hand, we used the tsfresh library [22] for timeseries feature extraction. Tsfresh provided a wide range of feature extraction methods tailored for time-series data, which helped capture complex temporal patterns of data. Finally, historical data was incorporated by including previous ALSFRS-R scores as features to provide historical context and improve model accuracy. This approach leverages the temporal aspect of the data, acknowledging that past scores are indicative of future outcomes.
The initial feature extraction process generated a vast number of features. To enhance model performance and reduce computational complexity, feature selection was performed based on relevance and significance.
Correlation analysis was conducted to analyze the correlation of features with target variables. Highly correlated features were prioritized as they are more likely to have a significant impact on model performance.
Preliminary models were trained to evaluate the importance of each feature. This approach helped identify which features contributed most to the predictive accuracy of the models. Features with low importance scores were discarded to streamline the model and improve eficiency.
The choice of a machine learning algorithm was crucial for the success of our predictive models. We ultimately selected the Random Forest Classifier (RFC) for its robustness and superior performance in handling multi-output regression tasks, very important for predicting multiple ALSFRS-R scores simultaneously. Initially, we considered several algorithms, including linear regression, support vector machines (SVM), and gradient boosting machines. The RFC’s ability to handle missing labels in classification further justified our choice. As an ensemble learning method, the RFC constructs multiple decision trees during training and averages their predictions. This approach is particularly advantageous for managing large datasets with high dimensionality and is resistant to overfitting.
To manage the prediction of multiple ALSFRS-R scores, a Multi-output Classifier was used. This metaestimator fitted a separate Random Forest Classifier for each target variable, allowing for simultaneous prediction of all 12 ALSFRS-R score components. By doing so, we ensured that each target is treated independently, which is beneficial given the varying degrees of correlation between diferent ALSFRS-R components.
A comprehensive data processing and modeling pipeline was constructed to streamline the workflow and ensure reproducibility. This pipeline was designed to handle all necessary preprocessing steps before feeding the data into the classifier. The key components of the pipeline included:
Preprocessing steps involved normalization and encoding of features to prepare the data for modeling. For numerical features, the StandardScaler was used to scale them to have a mean of zero and a standard deviation of one. Scaling is crucial for algorithms like Random Forest that are sensitive to the scales of input features. For categorical features such as sex, the OneHotEncoder was applied, transforming them into a format suitable for machine learning algorithms.
The last step in the pipeline was the application of the Multi-Output Classifier with the RFC as the underlying estimator. This step involved fitting the model to the training data and using it to make predictions.
For this study, hyperparameter tuning was performed using Randomized Search, a method that allows for more eficient exploration of the hyperparameter space compared to an exhaustive grid search. Randomized Search CV samples a fixed number of parameter settings from the specified distributions and evaluates them using cross-validation.
The hyperparameters tuned included the number of estimators, which refers to the number of trees in the forest. This parameter influences the overall performance and stability of the predictions. Maximum depth, the maximum depth of the trees, controls the complexity of the model and helps prevent overfitting. Minimum samples split, the minimum number of samples required to split an internal node, impacts the model’s ability to generalize to new data. Minimum samples leaf, the minimum number of samples required to be at a leaf node, similarly afects the model’s complexity and generalization. Maximum features, the number of features to consider when looking for the best split, can significantly afect the model’s performance by reducing overfitting and improving generalization. Lastly, bootstrap, which determines whether bootstrap samples are used when building trees, afects the diversity of the trees in the forest.
To ensure the robustness of the model, first opted for a multi-label stratified shufle split was used for cross-validation. This method maintains the balance of multiple target variables across the training and validation folds, ensuring that each fold is representative of the overall data distribution. However, due to label misrepresentation, a simple multi-label shufle split was used in most cases.
The following tables summarize the distinct methodological (Table 1) and ensemble approaches (Table 2) utilized for predicting ALSFRS-R scores in Tasks 1 and 2 of the iDPP CLEF 2024 challenge. Employs the average of observed values from the sensor data to establish a baseline prediction, simplifying the model by reducing noise and variance in the data.
Uses the median of the sensor data to provide a robust prediction that is less sensitive to outliers, aiming to capture the central tendency of the ALSFRS-R scores.
Integrates a wider array of statistical measures (mean, median, standard deviation, min, max) from the sensor data to create a comprehensive profile for prediction.
Leverages advanced time-series analysis techniques to capture and utilize the temporal patterns in the sensor data, emphasizing changes over time in the ALSFRS-R scores.
Uses the minimum value among all model predictions for each score, aiming to provide a conservative estimate of the ALSFRS-R scores.
Calculates the average of predictions from all models, balancing the influence of each model to mitigate extreme predictions and smooth outliers.
Selects the maximum value from the predictions, reflecting an optimistic estimation which may capture potential over-performances in patient assessment scores.
Each method and ensemble strategy is designed to address specific characteristics of ALS progression data. These approaches range from basic statistical summaries to complex time-series models and integrated ensemble methods, providing a solution set for diverse data analysis solutions.
The data was split into training, validation, and test sets, ensuring temporal consistency and patientspecific segregation to prevent data leakage and ensure reliable evaluation. We first separated the train datasets into an 80-20 split, where 80% of the data was used for training and the remainder for validation. Upon receiving the test data, training was done using all data.
Experiments were conducted on a virtual machine with a Ubunto OS equipped with 32 GB of memory configurations to handle the computational demands of feature extraction and model training.
In this section, we present the results for Task 1 and Task 2 of the iDPP CLEF 2024 challenge, covering both the validation and test phases. For both tasks, the same methodological approaches—Mean, Median, More Metrics, and Temporal Analysis—were consistently applied. Additionally, for the test predictions, we employed an ensemble approach that combined the results of the Mean, Median, More Metrics, and Temporal Analysis methods. This ensemble approach utilized the minimum, average, and maximum values of each predicted score to enhance the robustness and accuracy of our final model outputs. This approach allowed us to directly compare the efectiveness of each methodology across diferent stages of the evaluation process and assess the robustness of our models under varying conditions inherent in clinically-assessed and self-assessed ALS Functional Rating Scale-Revised (ALSFRS-R) scores.
The validation results for Task 1 and Task 2 of the iDPP CLEF 2024 challenge, presented in Tables 3 and 4 respectively, illustrate the performance of four methodologies—Mean, Median, Combination of several metrics (More Metrics), and Temporal—across twelve parameters of the ALS Functional Rating Scale-Revised (ALSFRS-R). These methodologies are quantified through Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), providing insights into model accuracy and consistency across two distinct tasks: clinically-assessed and self-assessed ALSFRS-R scores.
In Task 1, Temporal Analysis demonstrates the most consistent superior performance, notably achieving the lowest MAE and RMSE in several quarters, exemplified by its notable performance in Q2 with an MAE of 0.0370 and RMSE of 0.1925. This method efectively leverages time-series data to capture the dynamic progression of ALS, indicating its robust capability to handle the complexities inherent in clinical data. Conversely, the combined metrics approach, which integrates a broader array of statistical features, shows comparable MAE but slightly less RMSE in some parameters.
The simpler statistical approaches of the Mean and Median methodologies exhibit higher error rates, especially in parameters associated with complex symptomatology like respiratory functions in later quarters. These methods, although computationally simpler, struggle with modeling nuanced changes over time, leading to reduced predictive reliability.
For Task 2, the challenges shift towards handling the greater subjectivity of self-assessment data, which introduces variability in reporting and perception by patients themselves. Here, the Temporal method again shows its strength in specific quarters but with increased variability across all parameters, reflecting the challenges of capturing subjective self-assessments over time. The combination of metrics (More Metrics) method provides slightly more stable results compared to the Temporal approach, though none of the methodologies achieve the lower error metrics seen in Task 1.
Both tasks reveal that performance disparities across diferent ALSFRS-R parameters highlight the complexities of predictive modeling in ALS. Lower MAEs and RMSEs in the initial parameters suggest clearer data patterns early in the disease progression, whereas increased errors in later quarters reflect the inherent challenges with more complex or sporadically reported symptoms.
The comparative analysis of the two tasks underscores the crucial role of sophisticated modeling techniques, particularly those capable of handling temporal dynamics, to enhance the accuracy of predictive models in ALS monitoring.
The global results of the participation in the challenge iDPP of CLEF 2024 are shown in Table 5 and Table 6. In both cases, the results of our team in the test dataset mirror the validation results. #
For Task 1, our most successful method, the _ 1_, employed an ensemble model that selected the maximum score from a set of predictions, achieving a Mean Absolute Error (MAE) of 0.25 and a Root Mean Square Error (RMSE) of 0.544324192. This method ranked fourth overall, suggesting that an optimistic perspective on ALS progression, where higher ALSFRS-R scores indicate better functional ability, may align more closely with certain patient trajectories.
Following this, the _ 1_ method, which incorporated advanced time-series analysis tools like tsfresh for feature extraction, demonstrated the ability to capture the dynamic aspects of ALS progression with an MAE of 0.333333333 and an RMSE of 0.606889042. This method which used the temporal patterns in the sensor data demonstrates the importance of observing these results over time in the classification of ALSFRS-R scores.
Conversely, our simpler approaches, specifically _ 1_ and _ 1_, which condensed predictions into median or mean values, resulted in less precise outcomes with MAEs and RMSEs around 0.40 and 0.70, respectively. These methods, while straightforward, struggled to model the complexities of ALS progression adequately, reflecting the challenges in using reductionist approaches for such multifaceted medical data.
On the opposite end of our ensemble spectrum, the _ 1_, which utilized the minimum scores from ensemble predictions, produced the least efective outcomes among our entries, with an MAE of 0.48015873 and an RMSE of 0.795525562. This approach is likely overly pessimistic and thus struggled to achieve good results.
Comparatively, the performance of _ 1__ closely approached that of the competition leaders, who all achieved an MAE of 0.202380952 and an RMSE of 0.491212504. Although slightly behind, the marginal diferences of 0.0476 in MAE and 0.0531 in RMSE from the top results demonstrate competitive capability and underscore the potential of our ensemble and temporal analysis strategies.
Moreover, these methods significantly outperformed the median competition results (MAE of 0.403489069 and RMSE of 0.707744917), marking a clear advantage over the average performance. The distinctions in both MAE and RMSE underline the efectiveness of our approach and highlight our method’s robustness compared to the broader field.
#
The best-performing method in Task 2 was also the _ 2__, achieving a MAE of 0.325757576 and a RMSE of 0.608658838. Although less accurate compared to its performance in Task 1, this method still demonstrated robustness. It ranked third among all entries, suggesting that maximizing predictions can efectively capture optimistic patient-reported outcomes, likely reflecting better self-perceived functional abilities.
Following closely, the _ 2_ method, which incorporated a broader array of metrics (mean, median, standard deviation, minimum and maximum values), posted an MAE of 0.371212121 and an RMSE of 0.654232661.
The variation in model performance between Tasks 1 and 2 can largely be attributed to the nature of the target variables. Task 1 focused on medically assigned ALSFRS-R scores, which are likely more consistent and standardized due to their clinical origin. In contrast, Task 2 dealt with self-assessment scores that can be more subjective and influenced by the patient’s perception and mood at the time of evaluation. This inherent subjectivity and variability in self-assessment could make modeling more challenging, potentially explaining why the same methods yielded diferent levels of accuracy across the tasks.
Moreover, patient self-assessment might not always align closely with clinical evaluations, leading to discrepancies that can afect the performance of models trained primarily with clinical score patterns in mind. This misalignment could particularly impact methods like the _ 1_, which performed significantly worse in Task 2.
Despite these challenges, the _ 2__ method’s performance was quite competitive when compared to the leading entries in Task 2, fcool and unipd, both of which achieved an MAE of 0.287878788 and an RMSE of 0.577431447. The diferences in MAE and RMSE were 0.038 and 0.031, respectively, indicating that even with the subjective nature of self-assessments, our method remained robust.
Furthermore, this method significantly outperformed the median and mean results across all teams, where the median MAE was 0.585532747 and the median RMSE was 0.896730391. This performance indicates not only the efectiveness of the ensemble max strategy in capturing the higher ends of patient self-reported outcomes but also its overall reliability in a more variably reported dataset.
This study has demonstrated the eficacy of using sensor data from smartwatches and mobile health applications to predict ALS disease progression, substantiated through our participation in the iDPP CLEF 2024’s Tasks 1 and 2. These tasks emphasize the potential of machine learning models, particularly the Random Forest Classifier within an ensemble framework, to forecast the ALS Functional Rating Scale-Revised (ALSFRS-R) scores both from a clinical and a patient-centered perspective.
Our findings reveal that ensemble methods, particularly the maximization strategy, were the most efective across both tasks, though with varying degrees of success. Task 1, focusing on clinically assigned ALSFRS-R scores, benefited from the objective consistency these scores typically maintain, thereby allowing our models to achieve a higher predictive accuracy. Conversely, Task 2 involved predicting patient self-assessment scores that inherently presented more variability and subjectivity, posing greater challenges for our predictive models. The performance dip in Task 2 illustrates the complexities and the nuanced diferences between patient-perceived symptoms and clinically evaluated symptoms.
These outcomes highlight the critical role of precise data handling and feature engineering in enhancing model performance. The integration of advanced time-series analysis tools and the strategic handling of missing data were crucial in developing models that could efectively interpret the complex nature of ALS progression. However, the discrepancy in performance between the two tasks also points to the need for models that can better accommodate the subjective variabilities inherent in self-assessments.
Future research should focus on refining the algorithms used in predictive modeling of ALS progression. This involves enhancing data preprocessing techniques and incorporating advanced machine learning frameworks that can better handle the variability and complexities of ALS data. Improving feature engineering is essential, particularly by integrating sophisticated temporal features such as rolling means and time-windowed aggregates, which can provide a deeper understanding of disease dynamics. Additionally, applying deep learning techniques, especially those tailored for time-series analysis like Long Short-Term Memory (LSTM) networks, could significantly enhance model performance. Incorporating anomaly detection for outlier identification and signal processing techniques will further refine the models’ predictive capabilities. Exploring hybrid models that combine machine learning lfexibility with rule-based logic could also ofer robust solutions by merging data-driven insights with clinical expertise, leading to more accurate and responsive predictive models for ALS progression.
This work has received support from the EU/EFPIA Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 806968. It also has received support from the EC under grant agreement 101081813, Genomic Data Infrastructure. Heuvel, Deep learning predictions of survival based on mri in amyotrophic lateral sclerosis, NeuroImage: Clinical 13 (2017) 361–369. [15] A. Sengur, Y. Akbulut, Y. Guo, V. Bajaj, Classification of amyotrophic lateral sclerosis disease based on convolutional neural network and reinforcement sample learning algorithm, Health information science and systems 5 (2017) 1–7. [16] B. Yin, M. Balvert, R. A. van der Spek, B. E. Dutilh, S. Bohte, J. Veldink, A. Schönhuth, Using the structure of genome data in the design of deep neural networks for predicting amyotrophic lateral sclerosis from genotype, Bioinformatics 35 (2019) i538–i547. [17] M. Müller, M. Gromicho, M. de Carvalho, S. C. Madeira, Explainable models of disease progression in als: Learning from longitudinal clinical data with recurrent neural networks and deep model explanation, Computer Methods and Programs in Biomedicine Update 1 (2021) 100018. [18] C. Pancotti, G. Birolo, C. Rollo, T. Sanavia, B. Di Camillo, U. Manera, A. Chiò, P. Fariselli, Deep learning methods to predict amyotrophic lateral sclerosis disease progression, Scientific reports 12 (2022) 13738. [19] Brainteaser, Scientific publications, https://brainteaser.health/resources/scientific-publications/, 2024. Accessed: 2024-05-23. [20] G. Faggioli, A. Guazzo, S. Marchesin, L. Menotti, I. Trescato, A. Helena, R. Bergamaschi, G. Birolo, P. Cavalla, A. Chiò, et al., Overview of idpp@ clef 2023: the intelligent disease progression prediction challenge, in: CEUR WORKSHOP PROCEEDINGS, volume 3497, CEUR-WS, 2023, pp. 1123–1164. [21] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825– 2830. [22] M. Christ, N. Braun, J. Neufer, A. W. Kempa-Liehr, Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package), Neurocomputing 307 (2018) 72–77.