Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease causing progressive loss of cognitive and motor functions. Due to limited understanding of its mechanisms, there is no cure. Prognosis is still crucial for the effective planning of symptom treatment, however, the heterogeneity in patient progression drives the need for precision medicine research. iDPP CLEF 2024 aims to develop novel methodologies for predicting ALS disease progression, enabling the community to combine efforts and improve current prognostic methods. This report discusses our participation in tasks 1 and 2, evaluating the impact of sensor data on improving the prediction of ALSFRS-R scores. The proposed methodology combines temporal summarization techniques (extracting relevant statistics from the sensors), feature selection and extraction methods, and state-of-the-art classifiers for each ALSFRS-R question independently. Results show that random forest models yield the best overall performance, and selecting the k-best features and biclustering were the best overall feature selection and extraction strategies for tasks 1 and 2, respectively.
Amyotrophic Lateral Sclerosis (ALS) is a devastating neurodegenerative disease characterized by the progressive degeneration of motor neurons, leading to muscle weakness, atrophy, and eventual paralysis [1]. The progression of ALS varies significantly among patients, with some experiencing rapid deterioration while others decline more slowly [2]. This variability complicates the ability to predict disease trajectory, making it challenging for clinicians to offer accurate prognoses and for patients to make informed decisions about their future care [3].
Traditionally, clinical assessments of ALS progression rely on periodic evaluations using scales like the ALS Functional Rating Scale-Revised (ALSFRS-R) [4]. Although essential, these assessments provide only snapshots of a patient's condition at discrete time points and can miss subtle but critical changes between visits. This intermittent data collection limits the ability to detect early signs of disease worsening and delays the implementation of necessary interventions.
Recent advancements in sensor technology present a promising solution to these limitations. Sensors can generate a rich, real-time dataset by continuously monitoring physiological parameters such as muscle activity, respiratory function, and movement patterns [5]. This continuous data capture offers a detailed and dynamic view of a patient's condition, potentially revealing early indicators of disease progression that would otherwise go unnoticed between clinical visits [6].
However, to fully understand and predict ALS progression, it is essential to complement sensor data with patients' self-assessment data [7]. Self-assessments provide critical insights into subjective symptoms such as pain, fatigue, and emotional well-being, which are not easily quantifiable through sensors alone. Integrating objective sensor data with subjective self-assessment data creates a comprehensive, multidimensional dataset encompassing measurable physical changes and the patient's lived disease experience [8].
In this context, within the iDPP CLEF 2024 challenge1 framework, we tackled Tasks 1 and 2, which target predicting the twelve scores of the ALSFRS-R from sensor data. Task 1 aims to predict the score assigned by the clinician at the second visit, while Task 2 targets the second patient's selfassessment score. This paper reports the work done to overcome this challenge. We approach this challenge as a multi-label, multi-class classification approach with high-dimensional data. To handle the longitudinal datasets, we consider a double-step approach that transforms the time series sensor data using statistics computed from a time window period. Additionally, we test two feature selection strategies (K-Best features in all sensors and K-Best features in each sensor) and one feature extraction strategy (Biclustering-based features). To classify the ALSFRS-R scores, we train several state-of-the-art classifiers for each question independently.
Sensor technology has gained significant traction in recent years for monitoring ALS patients. Wearable sensors, such as accelerometers and gyroscopes, have continuously monitored motor function, gait, and other physical activities [5,9,10,11]. Accelerometer studies demonstrated their effectiveness in capturing detailed movement patterns, providing valuable data for assessing motor decline in ALS patients [6,10,9]. Vieira et al. [12] developed a model targeting ALS progression prediction based on voice samples and accelerometer measurements from a four-year longitudinal dataset. This model was used to predict bulbar-related and limb-related ALSFRS-R scores. Straczkiewicz et al. [11] used wrist wearables and ALSFRS-R self-entries data to propose new measures to quantify the count and duration of upper limb movements.
In addition to sensor data, integrating patients' self-assessment data has proven beneficial in understanding ALS progression. Studies have shown that self-reported pain, fatigue, and quality of life measures can provide critical insights that complement objective sensor data [13,7].
Machine learning techniques have been increasingly applied to predict disease progression in ALS [14]. Predicting the progression of the functional domains (twelve questions) assessed by the well-known functional scale, the ALSFRS-R was also investigated by Gordon and Lerner [15]. They modeled a multiclass classifier using demographic, respiratory assessments, genetic data, and other dynamic data to predict the values of each ALSFRS-R question at the time of the last patient visit.
Subspace techniques, such as pattern mining, biclustering, and triclustering, discover local patterns with non-constant coherencies with potential for predictive tasks. Martins et al. [16] recently proposed combining itemset mining with sequential pattern mining to uncover disease presentation and progression patterns in ALS patients and utilize these patterns to forecast the need for NIV. In a similar approach with the same prognostic target, Matos et al. [17] suggested a classifier based on biclustering. Biclustering [18,19] was used to locate groups of patients with similar values in subsets of clinical features (biclusters), which were then combined with static data as features. Although promising, none of these methods considered the temporal relationship of features. Soares et al. [20] proposed BicTric, a classifier capable of learning predictive models from both static and temporal data using discriminative patterns obtained through biclustering and triclustering [21,22,23]. Recently, Soares et al. [24] enhanced BicTric with TCtriCluster, a triclustering algorithm incorporating temporal contiguity constraints. These approaches utilized temporal preprocessing with snapshots and the time windows method proposed by Carreiro et al. [25] to learn predictive models for various clinically relevant ALS endpoints.
Integrating multi-modal data sources, including sensor data, self-assessments, and traditional clinical metrics, has shown potential in providing a more comprehensive understanding of ALS progression. Johnson et al. [8] conducted a study combining wearable sensor data with patient-reported outcomes and clinical assessments, demonstrating that multi-modal data fusion could enhance predictive accuracy and offer deeper insights into disease dynamics.
The objective of Tasks 1 and 2 of the iDPP CLEF 2024 challenge is to predict the values of the ALSFRS-R sub-scores of a second evaluation, given the values of the first evaluation. This would imply a reduced set of training instances (52 patients, in total), so we decided to generalize the challenge to predict the ALSFRS-R sub-scores of any evaluation given a previous evaluation, resulting in 121 training instances for Task 1 and 220 instances for Task 2.
The dataset made available [26,27] with this challenge contains information on ALS patients comprising the following data: static (including demographic and clinical information), all the ALSFRS-R evaluations (comprising the scores of the 12 questions for each patient), and sensor data (collected from the sensors of a fitness smartwatch). Figure 1 illustrates the processing of the dataset. Tasks 1 and 2 face a significant hurdle due to the sensor dataset's high dimensionality, stemming from a large number of sensor features (90 in total) and the numerous time points (approximately 268 sensor records per patient). To address this issue, we used a two-step processing of the dataset: first, we extracted temporal statistics from the longitudinal datasets. Second, we used feature selection or extraction techniques to obtain a representation of the sensor statistics with smaller dimensionality.
We derived new features from the longitudinal sensor data for each evaluation using summarization techniques, consisting of statistical metrics such as mean, standard deviation, minimum and maximum values, and the first and last values of each feature (as in Branco et al. [28]). To avoid the bias introduced by considering the entire sensor data history, these metrics were computed within fixed time intervals, specifically considering the interval [𝑡 − 𝛿, 𝑡], where 𝑡 represents the day of the target appointment and 𝛿 is the number of days within the interval (set to 15 days for Task 1 and 7 days for Task 2). This computation resulted in 540 new sensor features (90 original sensor features × 6 statistical metrics).
Another issue encountered with the dataset was missing values, even after the aforementioned computations. To address this, various interpolation and imputation techniques were explored, with polynomial interpolation of degree 5 proving to be the most effective in minimizing variance decrease across the feature sets.
After the interpolation step, sensor features exhibiting zero or near-zero variance (less than 10 −5 ) were deemed uninformative and consequently removed. Furthermore, highly correlated sensor features within the same category (calories, steps, beat_to_beat, heart_rate, respiration, and SpO 2 ) were also eliminated to mitigate redundancy. The selection of features for removal was based on Pearson correlation, with a correlation threshold set at 0.95 (see Table 1).
The sensor statistics obtained from the previously discussed step are still high dimensional, as there are 340 features for Task 1 and 352 features for Task 2. Subsequently, we applied three techniques (two feature selection (i) and (ii), and one feature extraction (iii)) to reduce the dataset dimensionality:
The first two feature selection techniques are based on a k-best selection strategy. First, we selected the top 5 features for predicting each target question based on ANOVA F-value between labels and features. Predictions were then made using the set of highest-ranked sensor statistical features across all questions (All Sensors). Alternatively, a specialized prediction approach was also adopted wherein the top 5 features were selected independently for each ALSFRS-R question based on mutual information (Each Target) (see Table 2). These selections were made using the SelectKBest class of the sklearn.feature_selection Python module.
As an alternative to these aforementioned feature selection strategies, we used a feature extraction strategy based on biclustering to reduce the dataset dimensionality. Biclustering, the simultaneous clustering of rows and columns of a data matrix, has shown its ability to discover local patterns with non-constant coherencies in both descriptive and predictive learning tasks [21,18]. Our approach, illustrated in Figure 2, applies biclustering to the Patient×Sensor Feature training matrix to obtain the
Number of selected top-ranked features, by category. Predictions were made using the pairs of strategy-models of highest-ranked computed sensor features, based on the ANOVA F-value, across all questions (All Sensors). Additionally, a specialized prediction method was employed, wherein the top 5 features were independently selected for each ALSFRS-R question based on mutual information (Each Target).
biclusters, with the row pattern of each bicluster being computed as the mean value of each column.
Then, the Euclidean distance between each training (and test) sample and the row pattern of each bicluster is computed to obtain a reduced representation of the training (and test) set. We used an approach based on biclustering-computed features. First, we apply a biclustering algorithm to obtain a set of biclusters (sub-matrices) from the dataset. Second, we compute its row pattern for each bicluster. Finally, we compute the distance between each row of the dataset and each bicluster to obtain the new reduced dataset. To simplify the representation of this methodology, we illustrate the pattern of a bicluster by the mode of each column (instead of the mean value) and use the Manhattan distance between each row and bicluster instead of the Euclidean distance.
We considered Spectral Biclustering to mine the biclusters as implemented in scikit-learn [29,30]. The number of biclusters influences the number of features in the reduced dataset. In our approach, we tested values for the number of biclusters and selected the value that maximizes the number of non-trivial biclusters (biclusters with more than 2 rows and columns).
In this section, we discuss our classification methodology, as illustrated in Figure 3. The tasks at hand are multi-label, multi-class tasks, which add complexity to the standard modeling techniques. Furthering the difficulty, the labels, which are the ALSFRS-R questions, are not completely independent, as the sub-scores are correlated within the different domains (bulbar, fine motor / upper limb, gross motor /lower limb, and respiratory).
Despite this intricacy, we decided to simplify the task by separating them into independent multi-class problems, where a given patient ALSFRS-R evaluation and their sensor data are used to predict each sub-score individually. Despite not modeling the correlation between questions, we assume that the models could still connect a patient's condition in time with their ability to perform a single function. We train 12 models and combine their predictions to predict the full set of sub-scores.
We consider a set of well-known classifiers covering a diverse range of model types, using scikitlearn [29]: Logistic Regression (LR), Random Forest (RF), XGBoost, and Support Vector Machines (SVC). Each model undergoes a model-appropriate pre-processing if required, and the optimal hyperparameters are searched for, as will be described later on.
For questions that have a sufficient representation across each of the scale values (0 to 4), 2 we employ imblearn [31]'s implementation of SMOTE [32], to alleviate the issue of small training sample size.
It is common to scale the input data for linear models to avoid widely different magnitudes across features that can hurt learning and performance. We use a standard scaler for Logistic Regression and Support Vector Machines to scale the input data.
We optimize the models using the Mean Absolute Error metric, both as a loss function for the model optimization and as a hyperparameter optimization objective, which searches for the best hyperparameter optimization that yields better performance on the validation set. We use Optuna [33] for hyperparameter optimization, with the Tree-Structured Parzen Estimator algorithm (as a sampler), avoiding a grid search brute-force approach to more efficiently sweep the hyperparameter space (see Table 3 for hyperparameter range of each model). The best-performing model is then used for the submissions in the challenge.
To assess the generalization of trained models and to optimize hyperparameters, we split the provided dataset into two sets: a train set and a validation set. As the dataset is multi-label multi-class, regular kernel CategoricalDistribution( ["linear", "rbf", "poly", "sigmoid"]) stratified train test splits do not guarantee a representative proportion of each scale value for each question for both splits. We resort to a variant termed iterative stratified train test splitting [34,35], implemented in the scikit-multilearn package [36]. This method works by iteratively populating both splits and assigning data points at each step to the split that requires them the most to maintain balance. Ultimately, we ensure each split is as similar to the overall dataset as possible. We split the provided training set following a 70/30 ratio, with 70% becoming the training set and 30% the validation set.
All the experiments were run on a Desktop Computer with an AMD Ryzen 9 7950X 16-Core with 64GB of RAM and Ubuntu 22.04.2. The code was run using Python 3.10.11.
In this section, we cover the results obtained in Tasks 1 and 2 in the challenge, as reported and computed with the private test set made available by the lab organizers.
To examine the impact of our design choices on feature selection or extraction, we define an experimental space beyond the basic analysis of the challenge results. First, for each question, we select the best pair feature selection or extraction strategy and classification with the top-k (we consider 𝑘 = {1, 2, 3}) highest validation metric values for both Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) (see section 4.1). Next, to determine which feature selection or extraction performs best, we consider the mean RMSE and MAE across the four classifiers for each question (see section 4.2). Lastly, we will assess whether there is a significant advantage in using one classifier over another. Given that the classifiers are all different types, identifying specific model properties suited for this particular task could lead to improvements for each question (see section 4.3).
We conducted experiments to predict the ALSFRS-R questions of a subsequent assessment by combining the best models for each target question based on their validation set performance. Specifically, we submitted the three best-performing pairs for both Tasks (see Table 4).
Table 5 presents the results of the models trained in each feature selection or extraction strategy for predicting each target question, along with the global results (average RMSE and MAE values across all questions). For both Tasks 1 and 2, the best-performing combination of feature selection or extraction strategy and classification model (strategy-model pair) in the test set was the second-best strategy-model pair in predicting the ALSFRS-R questions in the validation set. This suggests that the training and validation sets used for optimizing and validating the classifiers were unsuitable for predicting the ALSFRS-R questions in the second evaluation. These sets included all evaluations made available for the challenge, leading the models to be trained for predicting the next evaluation rather than specifically the second evaluation.
In Task 1, two questions related to the bulbar domain, Q1 and Q2, and one respiratory question, Q11, were the easiest to predict (RMSE 0.309, MAE 0.095). Specifically, Q1 and Q11 were best predicted using the XGBoost classifier with the All Sensors (Best strategy-model pair) and Each Target (2nd best strategy-model) feature selection strategies, respectively. Question Q2 was best predicted using the RF classifier with the All Sensors strategy (3rd best strategy-model pair). In contrast, motor-related questions, Q7 (trunk domain) and Q9 (lower limb domain) had the highest prediction errors (RMSE 0.873, MAE 0.476).
For Task 2, questions Q11 and Q12 were correctly classified for all the evaluations (RMSE 0.000 and MAE 0.000). Both the questions used the RF classifier and the Biclustering strategy (2nd best strategy-model and Best strategy-model pair, respectively). Question Q11 was also correctly classified for all evaluations using the Each Target strategy (3rd best strategy-model pair). Conversely, Q4 had the most misclassified evaluations (RMSE 1.044, MAE 0.545).
Results of the submitted strategy-model pairs. Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) metrics for the three best strategy-model pairs presented in Table 4. The performance metrics are provided for each target question and averaged across all the 12 questions (Global).
As previously mentioned, one feature extraction and two feature selection strategies were assessed: biclustering and K-Best selection, both globally for all questions (All Sensors) and individually for each question (Each Target). Table 6 presents the average model performance in the test set for each ALSFRS-R question and feature selection or extraction method. Overall, no strategy clearly outperformed the others, with the metrics typically not differing much between models with the same target question. However, the preferred strategy does change with the target.
In Task 1, the best overall method was individual k-best selection, Each Target (RMSE 0.780, MAE 0.474). It gathered the best average metrics in 6 out of the 12 questions, followed by the biclustering approach (RMSE 0.815, MAE 0.515) with 4 questions. Notably, there may be a preferred strategy by domain: the All Sensors approach performed best in the trunk domain questions (Q6 and Q7), and Each Target yielded the best metrics in the lower limb domain (Q8 and Q9). However, this behavior does not seem to occur for the upper limb domain (Q4 and Q5). For the bulbar (Q1-Q3) and respiratory (Q10-Q12) areas, the Biclustering and Each Target approaches achieved the best performance in two of the three targets. The best average performance was obtained for Q11 (RMSE 0.361, MAE 0.131) and the worst for Q6 (RMSE 0.909, MAE 0.667) and Q9 (RMSE 0.934, MAE 0.560).
For Task 2, the best overall strategy was feature transformation through Biclustering (RMSE 0.805, MAE 0.483), with the best average metrics in 8 out of 12 targets. Compared to Task 1, there is more overlap in the outcome of the three strategies, and as such, the second best method (Each Target; RMSE 0.836, MAE 0.507) had the best average metrics in 5 questions. Also, unlike Task 1, there is no preferred strategy by domain, save for the respiratory questions (Q10-Q12) that are most easily predicted by biclustering-based models. The best average performance was attained in Q12 (RMSE 0.419, MAE 0.318) and the worst in Q10 (RMSE 1.191, MAE 0.818).
We conducted experiments to predict the ALSFRS-R questions in the second evaluation using four machine-learning classifiers -Logistic Regression (LR), Random Forests (RF), Support Vector Machine (SVC), and XGBoost (XGB). We optimized their hyperparameters and validated their performance on a validation set derived from the provided training set as described in section 3.3. In addition to these classifiers, we also submitted two naïve approaches: Last Observation Carried Forward (LOCF) and Majority Class. Table 7 present the performance results for each model in predicting each target question, along with the overall results (average RMSE and MAE values across all questions). Notably, the LOCF approach performed the best for both tasks, indicating minimal variability between the ALSFRS-R scores of the first and second evaluations. On the other hand, the Majority Class approach was the worst performer, with RMSE values of 1.092 for Task 1 and 1.471 for Task 2. A potential reason for the classifiers' overall poor performance is that they were trained to predict the next score rather than specifically the second score, making the models too general for this particular task.
Regarding Task 1, questions Q3 (bulbar domain) and Q10 (respiratory domain) had the lowest prediction error using the LOCF approach (RMSE 0.218, MAE 0.048). Conversely, question Q9 (lower limb domain) predictions were the poorest, with the best classifier being RF (RMSE 0.873, MAE 0.381).
For Task 2, the conclusions are similar to those of Section 4.1. Questions regarding the respiratory domain, Q11, and Q12, were correctly predicted for all the evaluations. Particularly, the LOCF approach correctly predicted all the scores of question Q11, and the LR and RF classifiers accurately predicted all scores for question Q12 (RMSE 0.000, MAE 0.000). The most misclassified question was Q4 (upper limb domain), with an RMSE of 1.044 and MAE of 0.545 using the best-performing model (LOCF).
In a fast-acting and debilitating disease like ALS, the ability to predict how it evolves can be critical for clinical decision-making and life-prolonging therapy administration. Thus, the collection of sensor data can be a valuable resource for improving prognosis prediction, as it provides continuous monitoring of the patient's physiological status. This information can complement the periodic clinical assessments and possibly hint at the imminent occurrence of critical events, such as needing ventilation support. Machine learning techniques allow for meaningful insight to be extracted from these large datasets, which can potentially improve the performance of current prognosis prediction approaches or lead to the development of new ones. In the iDPP CLEF 2024 challenge, the main goal was to predict the ALSFRS-R scores (both clinical and self-assessed) of a patient's second assessment, given the first assessment and the sensor records between evaluations.
Our methodology consisted of independent multi-class models, each predicting an ALSFRS-R question. Four classification models were tested: Logistic Regression, Random Forest, XGBoost, and Support Vector Machine. The sensor data was handled first by deriving static features from the longitudinal ones using summarization techniques, i.e., by calculating summary statistics within an observation window before the target date. Then, the feature set was reduced using three methods: K-Best selection across all questions, K-Best selection by question, and biclustering. These models were also compared to baseline approaches Last Observation Carried Forward (LOCF) and Majority Class.
In both tasks, Random Forest yielded the best overall results but did not outperform LOCF, save for a few individual questions. Additionally, there was no consensus regarding the best feature selection or extraction approach. Independent K-Best selection and Biclustering were the best overall methods in tasks 1 and 2, respectively. However, further research is needed to capture the temporal patterns of sensors to fully understand their potential in tracking disease progression as measured by ALSFRS-R scores.
This work was partially supported by Fundação para a Ciência e a Tecnologia (FCT) through project AIpALS ref. PTDC/CCI-CIF/4613/2020 (https://doi.org/10.54499/PTDC/CCI-CIF/4613/2020), LASIGE Research Unit, ref. UIDB/00408/2020 (https://doi.org/10.54499/UIDB/00408/2020) and ref. UIDP/00408/2020 (https://doi.org/10.54499/UIDP/00408/2020), and PhD Research Scholarships to RB (2022.10727.BD), DFS ref. 2020.05100.BD (https://doi.org/10.54499/2020.05100.BD) and ENC ref. 2021.07810.BD (https: //doi.org/10.54499/2021.07810.BD); and by BRAINTEASER project, which has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 101017598.