Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease causing progressive loss of cognitive and motor functions. Due to limited understanding of its mechanisms, there is no cure. Prognosis is still crucial for the efective planning of symptom treatment, however, the heterogeneity in patient progression drives the need for precision medicine research. iDPP $ CLEF 2024 aims to develop novel methodologies for predicting ALS disease progression, enabling the community to combine eforts and improve current prognostic methods. This report discusses our participation in tasks 1 and 2, evaluating the impact of sensor data on improving the prediction of ALSFRS-R scores. The proposed methodology combines temporal summarization techniques (extracting relevant statistics from the sensors), feature selection and extraction methods, and state-of-the-art classifiers for each ALSFRS-R question independently. Results show that random forest models yield the best overall performance, and selecting the k-best features and biclustering were the best overall feature selection and extraction strategies for tasks 1 and 2, respectively.
Amyotrophic Lateral Sclerosis (ALS) is a devastating neurodegenerative disease characterized by the
progressive degeneration of motor neurons, leading to muscle weakness, atrophy, and eventual
paralysis [
Traditionally, clinical assessments of ALS progression rely on periodic evaluations using scales
like the ALS Functional Rating Scale-Revised (ALSFRS-R) [
Recent advancements in sensor technology present a promising solution to these limitations. Sensors
can generate a rich, real-time dataset by continuously monitoring physiological parameters such as
muscle activity, respiratory function, and movement patterns [
However, to fully understand and predict ALS progression, it is essential to complement sensor data
with patients’ self-assessment data [
In this context, within the iDPP $ CLEF 2024 challenge1 framework, we tackled Tasks 1 and 2, which target predicting the twelve scores of the ALSFRS-R from sensor data. Task 1 aims to predict the score assigned by the clinician at the second visit, while Task 2 targets the second patient’s selfassessment score. This paper reports the work done to overcome this challenge. We approach this challenge as a multi-label, multi-class classification approach with high-dimensional data. To handle the longitudinal datasets, we consider a double-step approach that transforms the time series sensor data using statistics computed from a time window period. Additionally, we test two feature selection strategies (K-Best features in all sensors and K-Best features in each sensor) and one feature extraction strategy (Biclustering-based features). To classify the ALSFRS-R scores, we train several state-of-the-art classifiers for each question independently.
Sensor technology has gained significant traction in recent years for monitoring ALS patients. Wearable
sensors, such as accelerometers and gyroscopes, have continuously monitored motor function, gait,
and other physical activities [
In addition to sensor data, integrating patients’ self-assessment data has proven beneficial in
understanding ALS progression. Studies have shown that self-reported pain, fatigue, and quality of life
measures can provide critical insights that complement objective sensor data [
Machine learning techniques have been increasingly applied to predict disease progression in ALS [
Subspace techniques, such as pattern mining, biclustering, and triclustering, discover local patterns with non-constant coherencies with potential for predictive tasks. Martins et al. [16] recently proposed combining itemset mining with sequential pattern mining to uncover disease presentation and progression patterns in ALS patients and utilize these patterns to forecast the need for NIV. In a similar approach with the same prognostic target, Matos et al. [17] suggested a classifier based on biclustering. Biclustering [18, 19] was used to locate groups of patients with similar values in subsets of clinical features (biclusters), which were then combined with static data as features. Although promising, none of these methods considered the temporal relationship of features. Soares et al. [20] proposed BicTric, a classifier capable of learning predictive models from both static and temporal data using discriminative patterns obtained through biclustering and triclustering [21, 22, 23]. Recently, Soares et al. [24] enhanced BicTric with TCtriCluster, a triclustering algorithm incorporating temporal contiguity constraints. These approaches utilized temporal preprocessing with snapshots and the time windows method proposed by Carreiro et al. [25] to learn predictive models for various clinically relevant ALS endpoints.
Integrating multi-modal data sources, including sensor data, self-assessments, and traditional clinical
metrics, has shown potential in providing a more comprehensive understanding of ALS progression.
Johnson et al. [
The objective of Tasks 1 and 2 of the iDPP $ CLEF 2024 challenge is to predict the values of the ALSFRS-R sub-scores of a second evaluation, given the values of the first evaluation. This would imply a reduced set of training instances (52 patients, in total), so we decided to generalize the challenge to predict the ALSFRS-R sub-scores of any evaluation given a previous evaluation, resulting in 121 training instances for Task 1 and 220 instances for Task 2.
The dataset made available [26, 27] with this challenge contains information on ALS patients comprising the following data: static (including demographic and clinical information), all the ALSFRS-R evaluations (comprising the scores of the 12 questions for each patient), and sensor data (collected from the sensors of a fitness smartwatch). Figure 1 illustrates the processing of the dataset. rs o n e S
Time
Time Series Statistics Patient n ito Temporal luaa Statistics v E Temporal Statistics litEvunaa (epvAreaLvSluioFauRstSivo-iRsnits) + o n o i t a u l a v E
ALSFRS-R Scores Clinical Variables n ALSFRS-R itoa evaluations lvua (next visit) E
GROUND TRUTH ALSFRS-R Scores
Selection
K-Best (all sensors) K-Best (each target) Extraction
Biclustering-based A p p e n d
Y X
Tasks 1 and 2 face a significant hurdle due to the sensor dataset’s high dimensionality, stemming from a large number of sensor features (90 in total) and the numerous time points (approximately 268 sensor records per patient). To address this issue, we used a two-step processing of the dataset: first, we extracted temporal statistics from the longitudinal datasets. Second, we used feature selection or extraction techniques to obtain a representation of the sensor statistics with smaller dimensionality.
We derived new features from the longitudinal sensor data for each evaluation using summarization techniques, consisting of statistical metrics such as mean, standard deviation, minimum and maximum values, and the first and last values of each feature (as in Branco et al. [28]). To avoid the bias introduced by considering the entire sensor data history, these metrics were computed within fixed time intervals, specifically considering the interval [ − , ], where represents the day of the target appointment and is the number of days within the interval (set to 15 days for Task 1 and 7 days for Task 2). This computation resulted in 540 new sensor features (90 original sensor features × 6 statistical metrics).
Another issue encountered with the dataset was missing values, even after the aforementioned computations. To address this, various interpolation and imputation techniques were explored, with polynomial interpolation of degree 5 proving to be the most efective in minimizing variance decrease across the feature sets.
After the interpolation step, sensor features exhibiting zero or near-zero variance (less than 10− 5) were deemed uninformative and consequently removed. Furthermore, highly correlated sensor features within the same category (calories, steps, beat_to_beat, heart_rate, respiration, and SpO2) were also eliminated to mitigate redundancy. The selection of features for removal was based on Pearson correlation, with a correlation threshold set at 0.95 (see Table 1).
The sensor statistics obtained from the previously discussed step are still high dimensional, as there are 340 features for Task 1 and 352 features for Task 2. Subsequently, we applied three techniques (two feature selection (i) and (ii), and one feature extraction (iii)) to reduce the dataset dimensionality: (i) K-Best features in all sensors; (ii) K-Best features in each target; (iii) Biclustering-based features.
The first two feature selection techniques are based on a k-best selection strategy. First, we selected the top 5 features for predicting each target question based on ANOVA F-value between labels and features. Predictions were then made using the set of highest-ranked sensor statistical features across all questions (All Sensors). Alternatively, a specialized prediction approach was also adopted wherein the top 5 features were selected independently for each ALSFRS-R question based on mutual information (Each Target) (see Table 2). These selections were made using the SelectKBest class of the sklearn.feature_selection Python module.
As an alternative to these aforementioned feature selection strategies, we used a feature extraction strategy based on biclustering to reduce the dataset dimensionality. Biclustering, the simultaneous clustering of rows and columns of a data matrix, has shown its ability to discover local patterns with non-constant coherencies in both descriptive and predictive learning tasks [21, 18]. Our approach, illustrated in Figure 2, applies biclustering to the Patient× Sensor Feature training matrix to obtain the biclusters, with the row pattern of each bicluster being computed as the mean value of each column. Then, the Euclidean distance between each training (and test) sample and the row pattern of each bicluster is computed to obtain a reduced representation of the training (and test) set.
Original Dataset (With two biclusters) y1 y2 y3 y4 y5 y6
We considered Spectral Biclustering to mine the biclusters as implemented in scikit-learn [29, 30]. The number of biclusters influences the number of features in the reduced dataset. In our approach, we tested values for the number of biclusters and selected the value that maximizes the number of non-trivial biclusters (biclusters with more than 2 rows and columns).
In this section, we discuss our classification methodology, as illustrated in Figure 3. ianrT pSlit egtarT uQestion uQestion uQestion uQestion lCasifer MSOTE oFr oMdel uQestion lCasifer u(sing yHperamt pOtimzaon uQestion lCasifer learn [29]: Logistic Regression (LR), Random Forest (RF), XGBoost, and Support Vector Machines (SVC). Each model undergoes a model-appropriate pre-processing if required, and the optimal hyperparameters are searched for, as will be described later on.
For questions that have a suficient representation across each of the scale values (0 to 4), 2 we employ imblearn [31]’s implementation of SMOTE [32], to alleviate the issue of small training sample size.
It is common to scale the input data for linear models to avoid widely diferent magnitudes across features that can hurt learning and performance. We use a standard scaler for Logistic Regression and Support Vector Machines to scale the input data.
We optimize the models using the Mean Absolute Error metric, both as a loss function for the model optimization and as a hyperparameter optimization objective, which searches for the best hyperparameter optimization that yields better performance on the validation set. We use Optuna [33] for hyperparameter optimization, with the Tree-Structured Parzen Estimator algorithm (as a sampler), avoiding a grid search brute-force approach to more eficiently sweep the hyperparameter space (see submissions in the challenge.
To assess the generalization of trained models and to optimize hyperparameters, we split the provided dataset into two sets: a train set and a validation set. As the dataset is multi-label multi-class, regular 2Two questions in each task did not qualify, which were questions 11 and 12 for Task 1, and 3 and 11 for Task 2. stratified train test splits do not guarantee a representative proportion of each scale value for each question for both splits. We resort to a variant termed iterative stratified train test splitting [ 34, 35], implemented in the scikit-multilearn package [36]. This method works by iteratively populating both splits and assigning data points at each step to the split that requires them the most to maintain balance. Ultimately, we ensure each split is as similar to the overall dataset as possible. We split the provided training set following a 70/30 ratio, with 70% becoming the training set and 30% the validation set.
All the experiments were run on a Desktop Computer with an AMD Ryzen 9 7950X 16-Core with 64GB of RAM and Ubuntu 22.04.2. The code was run using Python 3.10.11.
In this section, we cover the results obtained in Tasks 1 and 2 in the challenge, as reported and computed with the private test set made available by the lab organizers.
To examine the impact of our design choices on feature selection or extraction, we define an experimental space beyond the basic analysis of the challenge results. First, for each question, we select the best pair feature selection or extraction strategy and classification model with the top-k (we consider = {1, 2, 3}) highest validation metric values for both Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) (see section 4.1). Next, to determine which feature selection or extraction performs best, we consider the mean RMSE and MAE across the four classifiers for each question (see section 4.2). Lastly, we will assess whether there is a significant advantage in using one classifier over another. Given that the classifiers are all diferent types, identifying specific model properties suited for this particular task could lead to improvements for each question (see section 4.3).
We conducted experiments to predict the ALSFRS-R questions of a subsequent assessment by combining the best models for each target question based on their validation set performance. Specifically, we submitted the three best-performing pairs for both Tasks (see Table 4).
Table 5 presents the results of the models trained in each feature selection or extraction strategy for predicting each target question, along with the global results (average RMSE and MAE values across all questions). For both Tasks 1 and 2, the best-performing combination of feature selection or extraction strategy and classification model (strategy-model pair) in the test set was the second-best strategy-model pair in predicting the ALSFRS-R questions in the validation set. This suggests that the training and validation sets used for optimizing and validating the classifiers were unsuitable for predicting the ALSFRS-R questions in the second evaluation. These sets included all evaluations made available for the challenge, leading the models to be trained for predicting the next evaluation rather than specifically the second evaluation.
In Task 1, two questions related to the bulbar domain, Q1 and Q2, and one respiratory question, Q11, were the easiest to predict (RMSE 0.309, MAE 0.095). Specifically, Q1 and Q11 were best predicted using the XGBoost classifier with the All Sensors (Best strategy-model pair) and Each Target (2nd best strategy-model) feature selection strategies, respectively. Question Q2 was best predicted using the RF classifier with the All Sensors strategy (3rd best strategy-model pair). In contrast, motor-related questions, Q7 (trunk domain) and Q9 (lower limb domain) had the highest prediction errors (RMSE 0.873, MAE 0.476).
For Task 2, questions Q11 and Q12 were correctly classified for all the evaluations (RMSE 0.000 and MAE 0.000). Both the questions used the RF classifier and the Biclustering strategy (2nd best strategy-model and Best strategy-model pair, respectively). Question Q11 was also correctly classified for all evaluations using the Each Target strategy (3rd best strategy-model pair). Conversely, Q4 had the most misclassified evaluations (RMSE 1.044, MAE 0.545).
As previously mentioned, one feature extraction and two feature selection strategies were assessed: biclustering and K-Best selection, both globally for all questions (All Sensors) and individually for each question (Each Target).
Table 6 presents the average model performance in the test set for each ALSFRS-R question and feature selection or extraction method. Overall, no strategy clearly outperformed the others, with the metrics typically not difering much between models with the same target question. However, the preferred strategy does change with the target.
In Task 1, the best overall method was individual k-best selection, Each Target (RMSE 0.780, MAE 0.474). It gathered the best average metrics in 6 out of the 12 questions, followed by the biclustering approach (RMSE 0.815, MAE 0.515) with 4 questions. Notably, there may be a preferred strategy by domain: the All Sensors approach performed best in the trunk domain questions (Q6 and Q7), and Each Target yielded the best metrics in the lower limb domain (Q8 and Q9). However, this behavior does not seem to occur for the upper limb domain (Q4 and Q5). For the bulbar (Q1-Q3) and respiratory (Q10-Q12) areas, the Biclustering and Each Target approaches achieved the best performance in two of the three targets. The best average performance was obtained for Q11 (RMSE 0.361, MAE 0.131) and the worst for Q6 (RMSE 0.909, MAE 0.667) and Q9 (RMSE 0.934, MAE 0.560).
For Task 2, the best overall strategy was feature transformation through Biclustering (RMSE 0.805, MAE 0.483), with the best average metrics in 8 out of 12 targets. Compared to Task 1, there is more overlap in the outcome of the three strategies, and as such, the second best method (Each Target; RMSE 0.836, MAE 0.507) had the best average metrics in 5 questions. Also, unlike Task 1, there is no preferred strategy by domain, save for the respiratory questions (Q10-Q12) that are most easily predicted by biclustering-based models. The best average performance was attained in Q12 (RMSE 0.419, MAE 0.318) and the worst in Q10 (RMSE 1.191, MAE 0.818).
We conducted experiments to predict the ALSFRS-R questions in the second evaluation using four machine-learning classifiers - Logistic Regression (LR), Random Forests (RF), Support Vector Machine (SVC), and XGBoost (XGB). We optimized their hyperparameters and validated their performance on a validation set derived from the provided training set as described in section 3.3. In addition to these classifiers, we also submitted two naïve approaches: Last Observation Carried Forward (LOCF) and Majority Class.
In a fast-acting and debilitating disease like ALS, the ability to predict how it evolves can be critical for clinical decision-making and life-prolonging therapy administration. Thus, the collection of sensor data can be a valuable resource for improving prognosis prediction, as it provides continuous monitoring of the patient’s physiological status. This information can complement the periodic clinical assessments and possibly hint at the imminent occurrence of critical events, such as needing ventilation support. Machine learning techniques allow for meaningful insight to be extracted from these large datasets, which can potentially improve the performance of current prognosis prediction approaches or lead to the development of new ones. In the iDPP $ CLEF 2024 challenge, the main goal was to predict the ALSFRS-R scores (both clinical and self-assessed) of a patient’s second assessment, given the first assessment and the sensor records between evaluations.
Our methodology consisted of independent multi-class models, each predicting an ALSFRS-R question. Four classification models were tested: Logistic Regression, Random Forest, XGBoost, and Support Vector Machine. The sensor data was handled first by deriving static features from the longitudinal ones using summarization techniques, i.e., by calculating summary statistics within an observation window before the target date. Then, the feature set was reduced using three methods: K-Best selection across all questions, K-Best selection by question, and biclustering. These models were also compared to baseline approaches Last Observation Carried Forward (LOCF) and Majority Class.
In both tasks, Random Forest yielded the best overall results but did not outperform LOCF, save for a few individual questions. Additionally, there was no consensus regarding the best feature selection or extraction approach. Independent K-Best selection and Biclustering were the best overall methods in tasks 1 and 2, respectively. However, further research is needed to capture the temporal patterns of sensors to fully understand their potential in tracking disease progression as measured by ALSFRS-R scores.
This work was partially supported by Fundação para a Ciência e a Tecnologia (FCT) through project AIpALS ref. PTDC/CCI-CIF/4613/2020 (https://doi.org/10.54499/PTDC/CCI-CIF/4613/2020), LASIGE Research Unit, ref. UIDB/00408/2020 (https://doi.org/10.54499/UIDB/00408/2020) and ref. UIDP/00408/2020 (https://doi.org/10.54499/UIDP/00408/2020), and PhD Research Scholarships to RB (2022.10727.BD), DFS ref. 2020.05100.BD (https://doi.org/10.54499/2020.05100.BD) and ENC ref. 2021.07810.BD (https: //doi.org/10.54499/2021.07810.BD); and by BRAINTEASER project, which has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 101017598. [15] J. Gordon, B. Lerner, Insights into amyotrophic lateral sclerosis from a machine learning perspective,
Journal of Clinical Medicine 8 (2019) 1578. [16] A. S. Martins, M. Gromicho, S. Pinto, M. de Carvalho, S. C. Madeira, Learning prognostic models using diseaseprogression patterns: Predicting the need fornon-invasive ventilation in amyotrophic lateralsclerosis, IEEE/ACM Transactions on Computational Biology and Bioinformatics (2021). [17] J. Matos, S. Pires, H. Aidos, M. Gromicho, S. Pinto, M. de Carvalho, S. C. Madeira, Unravelling disease presentation patterns in als using biclustering for discriminative meta-features discovery, in: International Work-Conference on Bioinformatics and Biomedical Engineering, Springer, 2020, pp. 517–528. [18] S. C. Madeira, A. L. Oliveira, Biclustering algorithms for biological data analysis: a survey,
IEEE/ACM transactions on computational biology and bioinformatics 1 (2004) 24–45. [19] E. N. Castanho, H. Aidos, S. C. Madeira, Biclustering fmri time series: a comparative study, BMC bioinformatics 23 (2022) 192. [20] D. F. Soares, R. Henriques, M. Gromicho, M. de Carvalho, S. C. Madeira, Learning prognostic models using a mixture of biclustering and triclustering: Predicting the need for non-invasive ventilation in amyotrophic lateral sclerosis, Journal of Biomedical Informatics 134 (2022) 104172. [21] R. Henriques, S. C. Madeira, Flebic: Learning classifiers from high-dimensional biomedical data using discriminative biclusters with non-constant patterns, Pattern Recognition 115 (2021) 107900. [22] R. Henriques, S. C. Madeira, Triclustering algorithms for three-dimensional data analysis: a comprehensive survey, ACM Computing Surveys (CSUR) 51 (2018) 1–43. [23] D. F. Soares, R. Henriques, S. C. Madeira, Comprehensive assessment of triclustering algorithms for three-way temporal data analysis, Pattern Recognition (2024) 110303. [24] D. F. Soares, R. Henriques, M. Gromicho, M. de Carvalho, S. C. Madeira, Triclustering-based classification of longitudinal data for prognostic prediction: targeting relevant clinical endpoints in amyotrophic lateral sclerosis, Scientific Reports 13 (2023) 6182. [25] A. V. Carreiro, P. M. Amaral, S. Pinto, P. Tomás, M. de Carvalho, S. C. Madeira, Prognostic models based on patient snapshots and time windows: Predicting disease progression to assisted ventilation in amyotrophic lateral sclerosis, Journal of biomedical informatics 58 (2015) 133–144. [26] G. Birolo, P. Bosoni, G. Faggioli, H. Aidos, R. Bergamaschi, P. Cavalla, A. Chiò, A. Dagliati, M. de Carvalho, G. Di Nunzio, P. Fariselli, J. García Dominguez, A. G. Marta Gromicho, E. Longato, S. Madeira, U. Manera, S. Marchesin, L. Menotti, G. Silvello, E. Tavazzi, E. Tavazzi, I. Trescato, M. Vettoretti, B. D. Camillo, N. Ferro, Overview of iDPP@CLEF 2024: The Intelligent Disease Progression Prediction Challenge, in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, September 9th to 12th, 2024, 2024. [27] G. Birolo, P. Bosoni, G. Faggioli, H. Aidos, R. Bergamaschi, P. Cavalla, A. Chiò, A. Dagliati, M. de Carvalho, G. Di Nunzio, P. Fariselli, J. García Dominguez, A. G. Marta Gromicho, E. Longato, S. Madeira, U. Manera, S. Marchesin, L. Menotti, G. Silvello, E. Tavazzi, E. Tavazzi, I. Trescato, M. Vettoretti, B. D. Camillo, N. Ferro, Intelligent Disease Progression Prediction: Overview of iDPP@CLEF 2024, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction 15th International Conference of the CLEF Association, CLEF 2024, Grenoble, France, September 9-12, 2024, Proceedings, 2024. [28] R. Branco, J. Valente, A. Martins, D. Soares, E. Castanho, S. Madeira, H. Aidos, Survival analysis for multiple sclerosis: predicting risk of disease worsening, in: CLEF, 2023. [29] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825– 2830. [30] Y. Kluger, R. Basri, J. T. Chang, M. Gerstein, Spectral biclustering of microarray data: Coclustering genes and conditions, Genome Research 13 (2003) 703–716. doi:10.1101/gr.648603. [31] G. Lemaître, F. Nogueira, C. K. Aridas, Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning, Journal of Machine Learning Research 18 (2017) 1–5.
URL: http://jmlr.org/papers/v18/16-365.html. [32] N. V. Chawla, K. W. Bowyer, L. O. Hall, W. P. Kegelmeyer, Smote: synthetic minority over-sampling technique, Journal of artificial intelligence research 16 (2002) 321–357. [33] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019. [34] K. Sechidis, G. Tsoumakas, I. Vlahavas, On the stratification of multi-label data, Machine Learning and Knowledge Discovery in Databases (2011) 145–158. [35] P. Szymański, T. Kajdanowicz, A network perspective on stratification of multi-label data, in: L. Torgo, B. Krawczyk, P. Branco, N. Moniz (Eds.), Proceedings of the First International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 74 of Proceedings of Machine Learning Research, PMLR, ECML-PKDD, Skopje, Macedonia, 2017, pp. 22–35. [36] P. Szymański, T. Kajdanowicz, A scikit-based Python environment for performing multi-label classification, ArXiv e-prints (2017). arXiv:1702.01460.