Predicting the Functional Rating Scale and Self-Assessment Status of ALS Patients with Sensor Data Notebook for the iDPP@CLEF Lab at CLEF 2024 Andreia S. Martins† , Daniela M. Amaral† , Eduardo N. Castanho† , Diogo F. Soares, Ruben Branco* , Sara C. Madeira and Helena Aidos LASIGE, Faculdade de Ciências, Universidade de Lisboa, Portugal Abstract Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease causing progressive loss of cognitive and motor functions. Due to limited understanding of its mechanisms, there is no cure. Prognosis is still crucial for the effective planning of symptom treatment, however, the heterogeneity in patient progression drives the need for precision medicine research. iDPP $ CLEF 2024 aims to develop novel methodologies for predicting ALS disease progression, enabling the community to combine efforts and improve current prognostic methods. This report discusses our participation in tasks 1 and 2, evaluating the impact of sensor data on improving the prediction of ALSFRS-R scores. The proposed methodology combines temporal summarization techniques (extracting relevant statistics from the sensors), feature selection and extraction methods, and state-of-the-art classifiers for each ALSFRS-R question independently. Results show that random forest models yield the best overall performance, and selecting the k-best features and biclustering were the best overall feature selection and extraction strategies for tasks 1 and 2, respectively. Keywords Amyotrophic Lateral Sclerosis, Prognostic Prediction, Time Series Data, Biclustering, Multi-Class Classification 1. Introduction Amyotrophic Lateral Sclerosis (ALS) is a devastating neurodegenerative disease characterized by the progressive degeneration of motor neurons, leading to muscle weakness, atrophy, and eventual paral- ysis [1]. The progression of ALS varies significantly among patients, with some experiencing rapid deterioration while others decline more slowly [2]. This variability complicates the ability to predict disease trajectory, making it challenging for clinicians to offer accurate prognoses and for patients to make informed decisions about their future care [3]. Traditionally, clinical assessments of ALS progression rely on periodic evaluations using scales like the ALS Functional Rating Scale-Revised (ALSFRS-R) [4]. Although essential, these assessments provide only snapshots of a patient’s condition at discrete time points and can miss subtle but critical changes between visits. This intermittent data collection limits the ability to detect early signs of disease worsening and delays the implementation of necessary interventions. Recent advancements in sensor technology present a promising solution to these limitations. Sensors can generate a rich, real-time dataset by continuously monitoring physiological parameters such as muscle activity, respiratory function, and movement patterns [5]. This continuous data capture offers a detailed and dynamic view of a patient’s condition, potentially revealing early indicators of disease progression that would otherwise go unnoticed between clinical visits [6]. However, to fully understand and predict ALS progression, it is essential to complement sensor data with patients’ self-assessment data [7]. Self-assessments provide critical insights into subjective symp- toms such as pain, fatigue, and emotional well-being, which are not easily quantifiable through sensors CLEF 2024: Conference and Labs of the Evaluation Forum, September 09–12, 2024, Grenoble, France * Corresponding author. † These authors contributed equally. $ asmartins@ciencias.ulisboa.pt (A. S. Martins); daniela.amaral@tecnico.ulisboa.pt (D. M. Amaral); ejcastanho@ciencias.ulisboa.pt (E. N. Castanho); dfsoares@ciencias.ulisboa.pt (D. F. Soares); rmbranco@ciencias.ulisboa.pt (R. Branco); sacmadeira@ciencias.ulisboa.pt (S. C. Madeira); haidos@ciencias.ulisboa.pt (H. Aidos) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings alone. Integrating objective sensor data with subjective self-assessment data creates a comprehensive, multidimensional dataset encompassing measurable physical changes and the patient’s lived disease experience [8]. In this context, within the iDPP $ CLEF 2024 challenge1 framework, we tackled Tasks 1 and 2, which target predicting the twelve scores of the ALSFRS-R from sensor data. Task 1 aims to predict the score assigned by the clinician at the second visit, while Task 2 targets the second patient’s self- assessment score. This paper reports the work done to overcome this challenge. We approach this challenge as a multi-label, multi-class classification approach with high-dimensional data. To handle the longitudinal datasets, we consider a double-step approach that transforms the time series sensor data using statistics computed from a time window period. Additionally, we test two feature selection strategies (K-Best features in all sensors and K-Best features in each sensor) and one feature extraction strategy (Biclustering-based features). To classify the ALSFRS-R scores, we train several state-of-the-art classifiers for each question independently. 2. Related Work Sensor technology has gained significant traction in recent years for monitoring ALS patients. Wearable sensors, such as accelerometers and gyroscopes, have continuously monitored motor function, gait, and other physical activities [5, 9, 10, 11]. Accelerometer studies demonstrated their effectiveness in capturing detailed movement patterns, providing valuable data for assessing motor decline in ALS patients [6, 10, 9]. Vieira et al. [12] developed a model targeting ALS progression prediction based on voice samples and accelerometer measurements from a four-year longitudinal dataset. This model was used to predict bulbar-related and limb-related ALSFRS-R scores. Straczkiewicz et al. [11] used wrist wearables and ALSFRS-R self-entries data to propose new measures to quantify the count and duration of upper limb movements. In addition to sensor data, integrating patients’ self-assessment data has proven beneficial in un- derstanding ALS progression. Studies have shown that self-reported pain, fatigue, and quality of life measures can provide critical insights that complement objective sensor data [13, 7]. Machine learning techniques have been increasingly applied to predict disease progression in ALS [14]. Predicting the progression of the functional domains (twelve questions) assessed by the well-known functional scale, the ALSFRS-R was also investigated by Gordon and Lerner [15]. They modeled a multiclass classifier using demographic, respiratory assessments, genetic data, and other dynamic data to predict the values of each ALSFRS-R question at the time of the last patient visit. Subspace techniques, such as pattern mining, biclustering, and triclustering, discover local patterns with non-constant coherencies with potential for predictive tasks. Martins et al. [16] recently proposed combining itemset mining with sequential pattern mining to uncover disease presentation and pro- gression patterns in ALS patients and utilize these patterns to forecast the need for NIV. In a similar approach with the same prognostic target, Matos et al. [17] suggested a classifier based on biclustering. Biclustering [18, 19] was used to locate groups of patients with similar values in subsets of clinical features (biclusters), which were then combined with static data as features. Although promising, none of these methods considered the temporal relationship of features. Soares et al. [20] proposed BicTric, a classifier capable of learning predictive models from both static and temporal data using discriminative patterns obtained through biclustering and triclustering [21, 22, 23]. Recently, Soares et al. [24] enhanced BicTric with TCtriCluster, a triclustering algorithm incorporating temporal contiguity constraints. These approaches utilized temporal preprocessing with snapshots and the time windows method proposed by Carreiro et al. [25] to learn predictive models for various clinically relevant ALS endpoints. Integrating multi-modal data sources, including sensor data, self-assessments, and traditional clinical metrics, has shown potential in providing a more comprehensive understanding of ALS progression. Johnson et al. [8] conducted a study combining wearable sensor data with patient-reported outcomes 1 http://brainteaser.dei.unipd.it/challenges/idpp2024/ and clinical assessments, demonstrating that multi-modal data fusion could enhance predictive accuracy and offer deeper insights into disease dynamics. 3. Methodology The objective of Tasks 1 and 2 of the iDPP $ CLEF 2024 challenge is to predict the values of the ALSFRS-R sub-scores of a second evaluation, given the values of the first evaluation. This would imply a reduced set of training instances (52 patients, in total), so we decided to generalize the challenge to predict the ALSFRS-R sub-scores of any evaluation given a previous evaluation, resulting in 121 training instances for Task 1 and 220 instances for Task 2. The dataset made available [26, 27] with this challenge contains information on ALS patients com- prising the following data: static (including demographic and clinical information), all the ALSFRS-R evaluations (comprising the scores of the 12 questions for each patient), and sensor data (collected from the sensors of a fitness smartwatch). Figure 1 illustrates the processing of the dataset. Feature Selection/Extraction Selection K-Best (all sensors) Evaluation Sensor Temporal Sensor Time Series Statistics K-Best (each target) Data Statistics Extraction Biclustering-based nt tie Time Temporal Statistics Pa Append Evaluation Evaluation ALSFRS-R Static evaluations + Clinical (previous visit) Data Final Dataset ALSFRS-R Scores Clinical Variables ALSFRS-R Evaluation evaluations Y X (next visit) Classification GROUND TRUTH Pipeline ALSFRS-R Scores Predictions Figure 1: Data processing pipeline. Addressing the challenge implies handling data from three sources: static clinical variables, ALSFRS-R scores, and sensor data. To handle the highly dimensional sensor time series data, we computed statistics for each sensor and then applied feature selection or extraction strategies to reduce the dimensionality of the sensor dataset. The final dataset (that feeds the classifiers) aggregates these data sources. Tasks 1 and 2 face a significant hurdle due to the sensor dataset’s high dimensionality, stemming from a large number of sensor features (90 in total) and the numerous time points (approximately 268 sensor records per patient). To address this issue, we used a two-step processing of the dataset: first, we extracted temporal statistics from the longitudinal datasets. Second, we used feature selection or extraction techniques to obtain a representation of the sensor statistics with smaller dimensionality. 3.1. Time Series Statistics We derived new features from the longitudinal sensor data for each evaluation using summarization techniques, consisting of statistical metrics such as mean, standard deviation, minimum and maximum Table 1 Number of excluded sensor features, by category. The sensor data can be grouped into 6 distinct categories (Category). For each original sensor feature within these categories, 6 statistical metrics - mean, standard deviation, minimum value, maximum value, first value, and last value - were computed (#Computed Features). Features exhibiting zero or near-zero variance (#Low Variance) and those highly correlated with other features within the same category (#High Correlation) were removed from the dataset. Task 1 Task 2 Category #Computed Features #Low Variance #High Correlation #Low Variance #High Correlation calories 18 0 10 0 9 steps 24 0 3 0 3 beat_to_beat 240 13 116 10 108 heart_rate 60 10 1 9 2 respiration 108 0 27 0 31 SpO2 90 0 20 0 16 values, and the first and last values of each feature (as in Branco et al. [28]). To avoid the bias introduced by considering the entire sensor data history, these metrics were computed within fixed time intervals, specifically considering the interval [𝑡 − 𝛿, 𝑡], where 𝑡 represents the day of the target appointment and 𝛿 is the number of days within the interval (set to 15 days for Task 1 and 7 days for Task 2). This computation resulted in 540 new sensor features (90 original sensor features × 6 statistical metrics). Another issue encountered with the dataset was missing values, even after the aforementioned computations. To address this, various interpolation and imputation techniques were explored, with polynomial interpolation of degree 5 proving to be the most effective in minimizing variance decrease across the feature sets. After the interpolation step, sensor features exhibiting zero or near-zero variance (less than 10−5 ) were deemed uninformative and consequently removed. Furthermore, highly correlated sensor fea- tures within the same category (calories, steps, beat_to_beat, heart_rate, respiration, and SpO2 ) were also eliminated to mitigate redundancy. The selection of features for removal was based on Pearson correlation, with a correlation threshold set at 0.95 (see Table 1). 3.2. Feature Selection and Extraction Techniques The sensor statistics obtained from the previously discussed step are still high dimensional, as there are 340 features for Task 1 and 352 features for Task 2. Subsequently, we applied three techniques (two feature selection (i) and (ii), and one feature extraction (iii)) to reduce the dataset dimensionality: (i) K-Best features in all sensors; (ii) K-Best features in each target; (iii) Biclustering-based features. The first two feature selection techniques are based on a k-best selection strategy. First, we selected the top 5 features for predicting each target question based on ANOVA F-value between labels and features. Predictions were then made using the set of highest-ranked sensor statistical features across all questions (All Sensors). Alternatively, a specialized prediction approach was also adopted wherein the top 5 features were selected independently for each ALSFRS-R question based on mutual infor- mation (Each Target) (see Table 2). These selections were made using the SelectKBest class of the sklearn.feature_selection Python module. As an alternative to these aforementioned feature selection strategies, we used a feature extraction strategy based on biclustering to reduce the dataset dimensionality. Biclustering, the simultaneous clustering of rows and columns of a data matrix, has shown its ability to discover local patterns with non-constant coherencies in both descriptive and predictive learning tasks [21, 18]. Our approach, illustrated in Figure 2, applies biclustering to the Patient×Sensor Feature training matrix to obtain the Table 2 Number of selected top-ranked features, by category. Predictions were made using the pairs of strategy-models of highest-ranked computed sensor features, based on the ANOVA F-value, across all questions (All Sensors). Additionally, a specialized prediction method was employed, wherein the top 5 features were independently selected for each ALSFRS-R question based on mutual information (Each Target). Task 1 Task 2 calories steps beat_to_beat heart_rate respiration SpO2 calories steps beat_to_beat heart_rate respiration SpO2 n=8 n = 21 n = 111 n = 49 n = 81 n = 70 n=9 n = 21 n = 122 n = 49 n = 77 n = 74 All Sensors 6 6 13 5 3 3 6 5 19 1 5 5 Q1 3 0 1 1 0 0 1 0 0 0 1 3 Q2 0 0 0 3 0 2 4 0 1 0 0 0 Q3 5 0 0 0 0 0 5 0 0 0 0 0 Q4 0 0 4 0 1 0 0 0 3 0 0 2 Q5 0 0 5 0 0 0 0 0 5 0 0 0 Each Target Q6 0 3 2 0 0 0 0 0 5 0 0 0 Q7 0 4 1 0 0 0 0 0 5 0 0 0 Q8 0 5 0 0 0 0 0 4 1 0 0 0 Q9 0 5 0 0 0 0 0 4 1 0 0 0 Q10 0 0 2 1 2 0 1 0 3 0 1 0 Q11 1 0 3 0 0 1 0 0 4 1 0 0 Q12 0 0 5 0 0 0 0 0 0 0 5 0 biclusters, with the row pattern of each bicluster being computed as the mean value of each column. Then, the Euclidean distance between each training (and test) sample and the row pattern of each bicluster is computed to obtain a reduced representation of the training (and test) set. Original Dataset (With two biclusters) Reduced Dataset y1 y2 y3 y4 y5 y6 P1 P2 x1 1 3 1 5 4 4 x1 8 0 x2 5 1 2 5 2 3 x2 6 4 x3 3 2 3 2 3 1 x3 1 9 Train Data Biclustering Mapping x4 2 3 1 5 1 4 x4 9 0 x5 4 2 4 3 3 1 Pattern x5 0 9 y2 y3 y5 y6 1 x6 5 2 4 2 2 1 x6 1 10 2 4 3 1 x7 1 2 4 4 3 1 x7 1 8 Pattern y2 y3 y4 y6 2 3 1 5 4 y1 y2 y3 y4 y5 y6 P1 P2 x8 2 2 4 3 3 1 x8 0 9 Test Data x9 5 1 3 2 2 1 Mapping x9 3 10 x10 5 3 1 5 1 4 x10 9 0 x11 5 3 1 5 2 2 x11 6 2 Figure 2: We used an approach based on biclustering-computed features. First, we apply a biclustering algorithm to obtain a set of biclusters (sub-matrices) from the dataset. Second, we compute its row pattern for each bicluster. Finally, we compute the distance between each row of the dataset and each bicluster to obtain the new reduced dataset. To simplify the representation of this methodology, we illustrate the pattern of a bicluster by the mode of each column (instead of the mean value) and use the Manhattan distance between each row and bicluster instead of the Euclidean distance. We considered Spectral Biclustering to mine the biclusters as implemented in scikit-learn [29, 30]. The number of biclusters influences the number of features in the reduced dataset. In our approach, we tested values for the number of biclusters and selected the value that maximizes the number of non-trivial biclusters (biclusters with more than 2 rows and columns). 3.3. Modeling and Hyperparameter Optimization In this section, we discuss our classification methodology, as illustrated in Figure 3. Target SMOTE Question 1 For every model Hyperparameter Model Specific Pre-Processing Optimization Question i (using Train & Validation set) Train Split Question 12 Question 1 Question i Question 12 Classifier Classifier Classifier Independently performed for each question Figure 3: The challenge implies a multi-label, multi-class tasks. To simplify the training, we train classifiers for each question independently. We use SMOTE to compensate for a lack of sufficient representation across each scale value when possible. We train several traditional classifiers for each question, optimized considering the mean absolute error. The tasks at hand are multi-label, multi-class tasks, which add complexity to the standard modeling techniques. Furthering the difficulty, the labels, which are the ALSFRS-R questions, are not completely independent, as the sub-scores are correlated within the different domains (bulbar, fine motor / upper limb, gross motor /lower limb, and respiratory). Despite this intricacy, we decided to simplify the task by separating them into independent multi-class problems, where a given patient ALSFRS-R evaluation and their sensor data are used to predict each sub-score individually. Despite not modeling the correlation between questions, we assume that the models could still connect a patient’s condition in time with their ability to perform a single function. We train 12 models and combine their predictions to predict the full set of sub-scores. We consider a set of well-known classifiers covering a diverse range of model types, using scikit- learn [29]: Logistic Regression (LR), Random Forest (RF), XGBoost, and Support Vector Machines (SVC). Each model undergoes a model-appropriate pre-processing if required, and the optimal hyperparameters are searched for, as will be described later on. For questions that have a sufficient representation across each of the scale values (0 to 4),2 we employ imblearn [31]’s implementation of SMOTE [32], to alleviate the issue of small training sample size. It is common to scale the input data for linear models to avoid widely different magnitudes across features that can hurt learning and performance. We use a standard scaler for Logistic Regression and Support Vector Machines to scale the input data. We optimize the models using the Mean Absolute Error metric, both as a loss function for the model optimization and as a hyperparameter optimization objective, which searches for the best hyperparameter optimization that yields better performance on the validation set. We use Optuna [33] for hyperparameter optimization, with the Tree-Structured Parzen Estimator algorithm (as a sampler), avoiding a grid search brute-force approach to more efficiently sweep the hyperparameter space (see Table 3 for hyperparameter range of each model). The best-performing model is then used for the submissions in the challenge. To assess the generalization of trained models and to optimize hyperparameters, we split the provided dataset into two sets: a train set and a validation set. As the dataset is multi-label multi-class, regular 2 Two questions in each task did not qualify, which were questions 11 and 12 for Task 1, and 3 and 11 for Task 2. Table 3 The hyperparameter space for each model. Int and Float Distributions describe a search space between two integers or floating values, whereas CategoricalDistribution specifies a set of discrete values. Model Hyperparameter Distribution Space n_estimators IntDistribution(100, 1000) XGBoost Classifier max_depth IntDistribution(1, 20) learning_rate FloatDistribution(0.01, 1) n_estimators IntDistribution(10, 1000) Random Forest max_depth IntDistribution(1, 20) Logistic Regression(max_iter=100000) C FloatDistribution(0.01, 10) C FloatDistribution(0.01, 10) SVC(max_iter=100000, cache_size=1000) gamma FloatDistribution(0.01, 10) CategoricalDistribution( kernel ["linear", "rbf", "poly", "sigmoid"]) stratified train test splits do not guarantee a representative proportion of each scale value for each question for both splits. We resort to a variant termed iterative stratified train test splitting [34, 35], implemented in the scikit-multilearn package [36]. This method works by iteratively populating both splits and assigning data points at each step to the split that requires them the most to maintain balance. Ultimately, we ensure each split is as similar to the overall dataset as possible. We split the provided training set following a 70/30 ratio, with 70% becoming the training set and 30% the validation set. All the experiments were run on a Desktop Computer with an AMD Ryzen 9 7950X 16-Core with 64GB of RAM and Ubuntu 22.04.2. The code was run using Python 3.10.11. 4. Results & Discussion In this section, we cover the results obtained in Tasks 1 and 2 in the challenge, as reported and computed with the private test set made available by the lab organizers. To examine the impact of our design choices on feature selection or extraction, we define an experi- mental space beyond the basic analysis of the challenge results. First, for each question, we select the best pair feature selection or extraction strategy and classification model with the top-k (we consider 𝑘 = {1, 2, 3}) highest validation metric values for both Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) (see section 4.1). Next, to determine which feature selection or extraction per- forms best, we consider the mean RMSE and MAE across the four classifiers for each question (see section 4.2). Lastly, we will assess whether there is a significant advantage in using one classifier over another. Given that the classifiers are all different types, identifying specific model properties suited for this particular task could lead to improvements for each question (see section 4.3). 4.1. Selecting the best combination of feature strategy and classification model We conducted experiments to predict the ALSFRS-R questions of a subsequent assessment by combining the best models for each target question based on their validation set performance. Specifically, we submitted the three best-performing pairs for both Tasks (see Table 4). Table 5 presents the results of the models trained in each feature selection or extraction strategy for predicting each target question, along with the global results (average RMSE and MAE values across all questions). For both Tasks 1 and 2, the best-performing combination of feature selection or extraction strategy and classification model (strategy-model pair) in the test set was the second-best strategy-model pair in predicting the ALSFRS-R questions in the validation set. This suggests that the training and validation sets used for optimizing and validating the classifiers were unsuitable for Table 4 Results on the validation set for each combination of feature selection or extraction strategy and classification model. RF stands for Random Forest, SVC for Support Vector Machine Classifier, and LR for Logistic Regression. Best pair 2nd best pair 3rd best pair Question Strategy Model Strategy Model Strategy Model Q1 All Sensors XGBoost Each Target XGBoost Each Target RF Q2 Each Target RF Biclustering RF All Sensors RF Q3 Biclustering RF Biclustering XGBoost All Sensors RF Q4 Each Target SVC All Sensors RF All Sensors SVC Q5 Each Target RF All Sensors XGBoost Each Target SVC Task 1 Q6 Biclustering XGBoost Biclustering RF All Sensors XGBoost Q7 Each Target SVC Each Target RF All Sensors XGBoost Q8 All Sensors SVC All Sensors LR All Sensors XGBoost Q9 Biclustering XGBoost Biclustering RF All Sensors RF Q10 All Sensors SVC Biclustering SVC Each Target LR Q11 Biclustering XGBoost Each Target XGBoost Biclustering RF Q12 All Sensors RF Biclustering RF All Sensors LR Q1 All Sensors SVC Biclustering RF Each Target RF Q2 Each Target XGBoost Biclustering SVC Each Target LR Q3 Biclustering XGBoost Biclustering SVC All Sensors XGBoost Q4 Each Target XGBoost All Sensors XGBoost All Sensors RF Q5 All Sensors RF Biclustering RF Each Target RF Task 2 Q6 All Sensors XGBoost Each Target RF All Sensors RF Q7 All Sensors RF Biclustering RF Each Target XGBoost Q8 Biclustering RF Each Target XGBoost Each Target RF Q9 Biclustering XGBoost Biclustering RF Biclustering SVC Q10 All Sensors SVC Each Target RF Each Target XGBoost Q11 All Sensors XGBoost Biclustering RF Each Target RF Q12 Biclustering RF All Sensors SVC Each Target RF predicting the ALSFRS-R questions in the second evaluation. These sets included all evaluations made available for the challenge, leading the models to be trained for predicting the next evaluation rather than specifically the second evaluation. In Task 1, two questions related to the bulbar domain, Q1 and Q2, and one respiratory question, Q11, were the easiest to predict (RMSE 0.309, MAE 0.095). Specifically, Q1 and Q11 were best predicted using the XGBoost classifier with the All Sensors (Best strategy-model pair) and Each Target (2nd best strategy-model) feature selection strategies, respectively. Question Q2 was best predicted using the RF classifier with the All Sensors strategy (3rd best strategy-model pair). In contrast, motor-related questions, Q7 (trunk domain) and Q9 (lower limb domain) had the highest prediction errors (RMSE 0.873, MAE 0.476). For Task 2, questions Q11 and Q12 were correctly classified for all the evaluations (RMSE 0.000 and MAE 0.000). Both the questions used the RF classifier and the Biclustering strategy (2nd best strategy-model and Best strategy-model pair, respectively). Question Q11 was also correctly classified for all evaluations using the Each Target strategy (3rd best strategy-model pair). Conversely, Q4 had the most misclassified evaluations (RMSE 1.044, MAE 0.545). Table 5 Results of the submitted strategy-model pairs. Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) metrics for the three best strategy-model pairs presented in Table 4. The performance metrics are provided for each target question and averaged across all the 12 questions (Global). Model Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Global Best strategy-model pair RMSE 0.309 0.577 0.436 0.900 0.655 0.900 1.113 0.655 1.291 0.756 0.378 0.577 0.712 MAE 0.095 0.238 0.190 0.524 0.429 0.619 0.857 0.429 0.810 0.381 0.143 0.238 0.413 Task 1 2nd best strategy-model pair RMSE 0.787 0.690 0.655 0.816 0.756 0.900 0.900 0.873 0.873 0.378 0.309 0.577 0.709 MAE 0.429 0.286 0.333 0.476 0.476 0.619 0.619 0.572 0.381 0.143 0.095 0.238 0.389 3rd best strategy-model pair RMSE 0.787 0.309 0.756 0.976 1.000 0.690 0.873 0.787 1.069 0.926 0.378 0.845 0.783 MAE 0.429 0.095 0.381 0.571 0.619 0.476 0.476 0.429 0.667 0.476 0.143 0.429 0.433 Best strategy-model pair RMSE 0.905 0.739 0.522 1.206 1.279 1.000 0.674 0.798 0.302 1.414 1.206 0.000 0.837 MAE 0.636 0.364 0.273 0.727 0.909 0.818 0.455 0.636 0.091 1.091 0.364 0.000 0.530 Task 2 2nd strategy-model pair RMSE 1.000 0.798 0.739 1.044 0.953 0.739 0.522 0.798 0.302 1.000 0.000 0.603 0.708 MAE 0.636 0.455 0.364 0.545 0.545 0.545 0.273 0.636 0.091 0.636 0.000 0.182 0.409 3rd best strategy-model pair RMSE 0.953 0.798 0.522 1.044 0.853 0.905 0.905 0.739 0.603 1.679 0.000 0.302 0.775 MAE 0.545 0.455 0.273 0.545 0.545 0.818 0.636 0.545 0.364 1.182 0.000 0.909 0.500 4.2. Feature Selection and Extraction Comparison As previously mentioned, one feature extraction and two feature selection strategies were assessed: biclustering and K-Best selection, both globally for all questions (All Sensors) and individually for each question (Each Target). Table 6 presents the average model performance in the test set for each ALSFRS-R question and feature selection or extraction method. Overall, no strategy clearly outperformed the others, with the metrics typically not differing much between models with the same target question. However, the preferred strategy does change with the target. In Task 1, the best overall method was individual k-best selection, Each Target (RMSE 0.780, MAE 0.474). It gathered the best average metrics in 6 out of the 12 questions, followed by the biclustering approach (RMSE 0.815, MAE 0.515) with 4 questions. Notably, there may be a preferred strategy by domain: the All Sensors approach performed best in the trunk domain questions (Q6 and Q7), and Each Target yielded the best metrics in the lower limb domain (Q8 and Q9). However, this behavior does not seem to occur for the upper limb domain (Q4 and Q5). For the bulbar (Q1-Q3) and respiratory (Q10-Q12) areas, the Biclustering and Each Target approaches achieved the best performance in two of the three targets. The best average performance was obtained for Q11 (RMSE 0.361, MAE 0.131) and the worst for Q6 (RMSE 0.909, MAE 0.667) and Q9 (RMSE 0.934, MAE 0.560). For Task 2, the best overall strategy was feature transformation through Biclustering (RMSE 0.805, MAE 0.483), with the best average metrics in 8 out of 12 targets. Compared to Task 1, there is more overlap in the outcome of the three strategies, and as such, the second best method (Each Target; RMSE 0.836, MAE 0.507) had the best average metrics in 5 questions. Also, unlike Task 1, there is no preferred strategy by domain, save for the respiratory questions (Q10-Q12) that are most easily predicted by biclustering-based models. The best average performance was attained in Q12 (RMSE 0.419, MAE 0.318) and the worst in Q10 (RMSE 1.191, MAE 0.818). 4.3. Model Comparison We conducted experiments to predict the ALSFRS-R questions in the second evaluation using four machine-learning classifiers - Logistic Regression (LR), Random Forests (RF), Support Vector Machine (SVC), and XGBoost (XGB). We optimized their hyperparameters and validated their performance on a validation set derived from the provided training set as described in section 3.3. In addition to these classifiers, we also submitted two naïve approaches: Last Observation Carried Forward (LOCF) and Majority Class. Table 6 Model results’ summary, by feature selection and extraction strategy. Presented Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) report to the average performance of the 4 tested classifiers (LR, RF, SVC, XGBoost), in the test set. The performance metrics are provided for each target question and averaged across all of the strategy’s models (Global), with the best outcome in bold. Strategy Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Global Biclustering RMSE 0.730 0.775 0.616 0.820 0.909 1.008 1.015 0.843 1.095 0.825 0.378 0.761 0.815 MAE 0.393 0.488 0.298 0.440 0.643 0.702 0.810 0.571 0.667 0.548 0.143 0.476 0.515 Task 1 All Sensors RMSE 0.744 0.836 0.883 0.975 0.804 0.909 0.884 0.733 1.205 0.842 0.477 0.849 0.845 MAE 0.417 0.571 0.488 0.560 0.536 0.667 0.571 0.429 0.821 0.548 0.190 0.536 0.528 Each Target RMSE 0.826 0.813 0.548 0.906 0.765 0.959 0.948 0.595 0.934 1.004 0.361 0.703 0.780 MAE 0.464 0.536 0.262 0.488 0.488 0.714 0.679 0.310 0.560 0.631 0.131 0.429 0.474 Biclustering RMSE 0.738 0.910 0.726 1.115 1.028 0.892 0.574 0.698 0.452 1.191 0.914 0.419 0.805 MAE 0.432 0.523 0.364 0.659 0.614 0.705 0.341 0.500 0.227 0.818 0.295 0.318 0.483 Task 2 All Sensors RMSE 0.820 0.799 0.749 1.217 1.310 1.034 0.811 0.797 0.689 1.388 1.383 0.603 0.967 MAE 0.545 0.477 0.432 0.727 0.977 0.864 0.554 0.500 0.386 0.977 0.568 0.364 0.614 Each Target RMSE 0.808 0.860 0.686 1.147 0.993 0.875 0.696 0.665 0.518 1.232 1.128 0.433 0.836 MAE 0.455 0.477 0.295 0.727 0.636 0.636 0.455 0.455 0.273 0.864 0.500 0.318 0.507 Table 7 present the performance results for each model in predicting each target question, along with the overall results (average RMSE and MAE values across all questions). Notably, the LOCF approach performed the best for both tasks, indicating minimal variability between the ALSFRS-R scores of the first and second evaluations. On the other hand, the Majority Class approach was the worst performer, with RMSE values of 1.092 for Task 1 and 1.471 for Task 2. A potential reason for the classifiers’ overall poor performance is that they were trained to predict the next score rather than specifically the second score, making the models too general for this particular task. Regarding Task 1, questions Q3 (bulbar domain) and Q10 (respiratory domain) had the lowest prediction error using the LOCF approach (RMSE 0.218, MAE 0.048). Conversely, question Q9 (lower limb domain) predictions were the poorest, with the best classifier being RF (RMSE 0.873, MAE 0.381). For Task 2, the conclusions are similar to those of Section 4.1. Questions regarding the respiratory domain, Q11, and Q12, were correctly predicted for all the evaluations. Particularly, the LOCF approach correctly predicted all the scores of question Q11, and the LR and RF classifiers accurately predicted all scores for question Q12 (RMSE 0.000, MAE 0.000). The most misclassified question was Q4 (upper limb domain), with an RMSE of 1.044 and MAE of 0.545 using the best-performing model (LOCF). 5. Conclusion In a fast-acting and debilitating disease like ALS, the ability to predict how it evolves can be critical for clinical decision-making and life-prolonging therapy administration. Thus, the collection of sensor data can be a valuable resource for improving prognosis prediction, as it provides continuous monitoring of the patient’s physiological status. This information can complement the periodic clinical assessments and possibly hint at the imminent occurrence of critical events, such as needing ventilation support. Machine learning techniques allow for meaningful insight to be extracted from these large datasets, which can potentially improve the performance of current prognosis prediction approaches or lead to the development of new ones. In the iDPP $ CLEF 2024 challenge, the main goal was to predict the ALSFRS-R scores (both clinical and self-assessed) of a patient’s second assessment, given the first assessment and the sensor records between evaluations. Our methodology consisted of independent multi-class models, each predicting an ALSFRS-R question. Four classification models were tested: Logistic Regression, Random Forest, XGBoost, and Support Table 7 Results of the models. Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) metrics of four ML classifiers and two naïve approaches across the 12 target questions. The classifiers include Logistic Regression (LR), Random Forest (RF), Support Vector Classifier (SVC), and XGBoost. The naïve approaches are the Last Observation Carried Forward (LOCF) and Majority Class. The performance metrics are provided for each target question and averaged across all the questions (Global). Model Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Global LOCF RMSE 0.488 0.309 0.218 0.690 0.535 0.577 0.488 0.535 0.951 0.218 0.309 0.577 0.491 MAE 0.143 0.095 0.048 0.286 0.286 0.333 0.238 0.190 0.429 0.048 0.095 0.238 0.202 Majority Class RMSE 1.512 0.976 1.512 1.254 1.113 1.34 1.327 1.175 1.690 0.309 0.378 0.724 1.092 MAE 0.857 0.476 0.762 0.814 0.762 0.905 0.810 0.810 1.238 0.0952 0.143 0.333 0.659 Task 1 LR RMSE 1.000 0.756 0.756 0.900 0.787 1.000 1.024 0.873 0.926 0.816 0.378 0.845 0.838 MAE 0.619 0.381 0.381 0.524 0.524 0.714 0.762 0.571 0.571 0.476 0.143 0.429 0.508 RF RMSE 0.690 0.578 0.436 0.926 0.655 0.900 0.900 0.617 0.873 0.577 0.378 0.577 0.676 MAE 0.381 0.238 0.190 0.476 0.429 0.619 0.619 0.286 0.381 0.238 0.143 0.238 0.353 SVC RMSE 0.976 0.787 0.617 0.900 1.000 1.234 1.113 0.655 1.291 0.756 0.378 0.951 0.888 MAE 0.571 0.429 0.286 0.524 0.619 0.857 0.857 0.429 0.905 0.381 0.143 0.524 0.544 XGBoost RMSE 0.309 1.134 0.655 0.900 0.617 0.900 0.756 0.787 1.291 1.215 0.378 1.024 0.830 MAE 0.095 1.095 0.333 0.429 0.381 0.619 0.476 0.429 0.810 1.095 0.143 0.952 0.571 LOCF RMSE 0.674 0.674 0.426 1.044 0.739 0.603 0.739 0.603 0.302 0.522 0.000 0.603 0.577 MAE 0.455 0.273 0.182 0.545 0.364 0.364 0.364 0.364 0.091 0.273 0.000 0.182 0.288 Majority Class RMSE 1.348 0.905 1.168 1.314 1.477 1.809 1.651 1.044 1.883 1.758 2.089 1.206 1.471 MAE 0.909 0.455 0.636 0.818 1.091 1.636 1.273 0.909 1.545 1.273 1.091 0.727 1.030 Task 2 LR RMSE 0.798 0.790 0.905 1.168 1.168 0.953 0.674 0.798 0.603 1.279 1.537 0.000 0.890 MAE 0.455 0.455 0.455 0.818 0.818 0.727 0.455 0.636 0.364 0.909 0.727 0.000 0.568 RF RMSE 0.905 0.905 0.739 1.087 1.279 0.905 0.674 0.798 0.302 1.128 1.508 0.000 0.852 MAE 0.636 0.455 0.364 0.636 0.909 0.818 0.455 0.636 0.091 0.727 0.636 0.000 0.530 SVC RMSE 0.905 1.000 0.739 1.128 1.624 1.279 0.853 0.674 0.603 1.414 1.279 0.674 1.014 MAE 0.636 0.636 0.364 0.727 1.364 1.091 0.545 0.455 0.364 1.091 0.545 0.273 0.674 XGBoost RMSE 0.674 0.739 0.522 1.206 1.168 1.000 1.044 0.522 0.302 1.732 1.206 1.000 0.926 MAE 0.455 0.364 0.273 0.727 0.818 0.818 0.727 0.273 0.091 1.182 0.364 1.000 0.591 Vector Machine. The sensor data was handled first by deriving static features from the longitudinal ones using summarization techniques, i.e., by calculating summary statistics within an observation window before the target date. Then, the feature set was reduced using three methods: K-Best selection across all questions, K-Best selection by question, and biclustering. These models were also compared to baseline approaches Last Observation Carried Forward (LOCF) and Majority Class. In both tasks, Random Forest yielded the best overall results but did not outperform LOCF, save for a few individual questions. Additionally, there was no consensus regarding the best feature selection or extraction approach. Independent K-Best selection and Biclustering were the best overall methods in tasks 1 and 2, respectively. However, further research is needed to capture the temporal patterns of sensors to fully understand their potential in tracking disease progression as measured by ALSFRS-R scores. Acknowledgments This work was partially supported by Fundação para a Ciência e a Tecnologia (FCT) through project AIpALS ref. PTDC/CCI-CIF/4613/2020 (https://doi.org/10.54499/PTDC/CCI-CIF/4613/2020), LASIGE Re- search Unit, ref. UIDB/00408/2020 (https://doi.org/10.54499/UIDB/00408/2020) and ref. UIDP/00408/2020 (https://doi.org/10.54499/UIDP/00408/2020), and PhD Research Scholarships to RB (2022.10727.BD), DFS ref. 2020.05100.BD (https://doi.org/10.54499/2020.05100.BD) and ENC ref. 2021.07810.BD (https: //doi.org/10.54499/2021.07810.BD); and by BRAINTEASER project, which has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 101017598. References [1] L. C. Wijesekera, P. Nigel Leigh, Amyotrophic lateral sclerosis, Orphanet journal of rare diseases 4 (2009) 1–22. [2] J. Morris, Amyotrophic lateral sclerosis (ALS) and related motor neuron diseases: an overview, The Neurodiagnostic Journal 55 (2015) 180–194. [3] S. R. Pfohl, R. B. Kim, G. S. Coan, C. S. Mitchell, Unraveling the complexity of amyotrophic lateral sclerosis survival prediction, Frontiers in neuroinformatics 12 (2018) 36. [4] J. M. Cedarbaum, N. Stambler, E. Malta, C. Fuller, D. Hilt, B. Thurmond, A. Nakanishi, B. A. S. Group, A. complete listing of the BDNF Study Group, et al., The ALSFRS-R: a revised als functional rating scale that incorporates assessments of respiratory function, Journal of the neurological sciences 169 (1999) 13–21. [5] E. Beswick, T. Fawcett, Z. Hassan, D. Forbes, R. Dakin, J. Newton, S. Abrahams, A. Carson, S. Chandran, D. Perry, et al., A systematic review of digital technology to evaluate motor function and disease progression in motor neuron disease, Journal of Neurology 269 (2022) 6254–6268. [6] R. P. van Eijk, J. N. Bakers, T. M. Bunte, A. J. de Fockert, M. J. Eijkemans, L. H. van den Berg, Accelerometry for remote monitoring of physical activity in amyotrophic lateral sclerosis: a longitudinal cohort study, Journal of neurology 266 (2019) 2387–2395. [7] A. Maier, T. Holm, P. Wicks, L. Steinfurth, P. Linke, C. Münch, R. Meyer, T. Meyer, Online assessment of als functional rating scale compares well to in-clinic evaluation: a prospective trial, Amyotrophic Lateral Sclerosis 13 (2012) 210–216. [8] S. A. Johnson, M. Karas, K. M. Burke, M. Straczkiewicz, Z. A. Scheier, A. P. Clark, S. Iwasaki, A. Lahav, A. S. Iyer, J.-P. Onnela, et al., Wearable device and smartphone data quantify als progression and may provide novel outcome measures, NPJ Digital Medicine 6 (2023) 34. [9] J. W. van Unnik, M. Meyjes, M. R. J. van Mantgem, L. H. van den Berg, R. P. van Eijk, Remote monitoring of amyotrophic lateral sclerosis using wearable sensors detects differences in disease progression and survival: a prospective cohort study, Ebiomedicine 103 (2024). [10] A. S. Gupta, S. Patel, A. Premasiri, F. Vieira, At-home wearables and machine learning sensitively capture disease progression in amyotrophic lateral sclerosis, Nature Communications 14 (2023) 5080. [11] M. Straczkiewicz, M. Karas, S. A. Johnson, K. M. Burke, Z. Scheier, T. B. Royse, N. Calcagno, A. Clark, A. Iyer, J. D. Berry, et al., Upper limb movements as digital biomarkers in people with als, EBioMedicine 101 (2024). [12] F. G. Vieira, S. Venugopalan, A. S. Premasiri, M. McNally, A. Jansen, K. McCloskey, M. P. Brenner, S. Perrin, A machine-learning based objective measure for als disease severity, NPJ digital medicine 5 (2022) 45. [13] S. B. Rutkove, P. Narayanaswami, V. Berisha, J. Liss, S. Hahn, K. Shelton, K. Qi, S. Pandeya, J. M. Shefner, Improved als clinical trials through frequent at-home self-assessment: a proof of concept study, Annals of Clinical and Translational Neurology 7 (2020) 1148–1157. [14] E. Tavazzi, E. Longato, M. Vettoretti, H. Aidos, I. Trescato, C. Roversi, A. S. Martins, E. N. Castanho, R. Branco, D. F. Soares, et al., Artificial intelligence and statistical methods for stratification and prediction of progression in amyotrophic lateral sclerosis: A systematic review, Artificial Intelligence in Medicine (2023) 102588. [15] J. Gordon, B. Lerner, Insights into amyotrophic lateral sclerosis from a machine learning perspective, Journal of Clinical Medicine 8 (2019) 1578. [16] A. S. Martins, M. Gromicho, S. Pinto, M. de Carvalho, S. C. Madeira, Learning prognostic models using diseaseprogression patterns: Predicting the need fornon-invasive ventilation in amyotrophic lateralsclerosis, IEEE/ACM Transactions on Computational Biology and Bioinformatics (2021). [17] J. Matos, S. Pires, H. Aidos, M. Gromicho, S. Pinto, M. de Carvalho, S. C. Madeira, Unravelling disease presentation patterns in als using biclustering for discriminative meta-features discovery, in: International Work-Conference on Bioinformatics and Biomedical Engineering, Springer, 2020, pp. 517–528. [18] S. C. Madeira, A. L. Oliveira, Biclustering algorithms for biological data analysis: a survey, IEEE/ACM transactions on computational biology and bioinformatics 1 (2004) 24–45. [19] E. N. Castanho, H. Aidos, S. C. Madeira, Biclustering fmri time series: a comparative study, BMC bioinformatics 23 (2022) 192. [20] D. F. Soares, R. Henriques, M. Gromicho, M. de Carvalho, S. C. Madeira, Learning prognostic models using a mixture of biclustering and triclustering: Predicting the need for non-invasive ventilation in amyotrophic lateral sclerosis, Journal of Biomedical Informatics 134 (2022) 104172. [21] R. Henriques, S. C. Madeira, Flebic: Learning classifiers from high-dimensional biomedical data using discriminative biclusters with non-constant patterns, Pattern Recognition 115 (2021) 107900. [22] R. Henriques, S. C. Madeira, Triclustering algorithms for three-dimensional data analysis: a comprehensive survey, ACM Computing Surveys (CSUR) 51 (2018) 1–43. [23] D. F. Soares, R. Henriques, S. C. Madeira, Comprehensive assessment of triclustering algorithms for three-way temporal data analysis, Pattern Recognition (2024) 110303. [24] D. F. Soares, R. Henriques, M. Gromicho, M. de Carvalho, S. C. Madeira, Triclustering-based classification of longitudinal data for prognostic prediction: targeting relevant clinical endpoints in amyotrophic lateral sclerosis, Scientific Reports 13 (2023) 6182. [25] A. V. Carreiro, P. M. Amaral, S. Pinto, P. Tomás, M. de Carvalho, S. C. Madeira, Prognostic models based on patient snapshots and time windows: Predicting disease progression to assisted ventilation in amyotrophic lateral sclerosis, Journal of biomedical informatics 58 (2015) 133–144. [26] G. Birolo, P. Bosoni, G. Faggioli, H. Aidos, R. Bergamaschi, P. Cavalla, A. Chiò, A. Dagliati, M. de Carvalho, G. Di Nunzio, P. Fariselli, J. García Dominguez, A. G. Marta Gromicho, E. Longato, S. Madeira, U. Manera, S. Marchesin, L. Menotti, G. Silvello, E. Tavazzi, E. Tavazzi, I. Trescato, M. Vettoretti, B. D. Camillo, N. Ferro, Overview of iDPP@CLEF 2024: The Intelligent Disease Progression Prediction Challenge, in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, September 9th to 12th, 2024, 2024. [27] G. Birolo, P. Bosoni, G. Faggioli, H. Aidos, R. Bergamaschi, P. Cavalla, A. Chiò, A. Dagliati, M. de Carvalho, G. Di Nunzio, P. Fariselli, J. García Dominguez, A. G. Marta Gromicho, E. Longato, S. Madeira, U. Manera, S. Marchesin, L. Menotti, G. Silvello, E. Tavazzi, E. Tavazzi, I. Trescato, M. Vettoretti, B. D. Camillo, N. Ferro, Intelligent Disease Progression Prediction: Overview of iDPP@CLEF 2024, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Grenoble, France, September 9-12, 2024, Proceedings, 2024. [28] R. Branco, J. Valente, A. Martins, D. Soares, E. Castanho, S. Madeira, H. Aidos, Survival analysis for multiple sclerosis: predicting risk of disease worsening, in: CLEF, 2023. [29] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825– 2830. [30] Y. Kluger, R. Basri, J. T. Chang, M. Gerstein, Spectral biclustering of microarray data: Coclustering genes and conditions, Genome Research 13 (2003) 703–716. doi:10.1101/gr.648603. [31] G. Lemaître, F. Nogueira, C. K. Aridas, Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning, Journal of Machine Learning Research 18 (2017) 1–5. URL: http://jmlr.org/papers/v18/16-365.html. [32] N. V. Chawla, K. W. Bowyer, L. O. Hall, W. P. Kegelmeyer, Smote: synthetic minority over-sampling technique, Journal of artificial intelligence research 16 (2002) 321–357. [33] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019. [34] K. Sechidis, G. Tsoumakas, I. Vlahavas, On the stratification of multi-label data, Machine Learning and Knowledge Discovery in Databases (2011) 145–158. [35] P. Szymański, T. Kajdanowicz, A network perspective on stratification of multi-label data, in: L. Torgo, B. Krawczyk, P. Branco, N. Moniz (Eds.), Proceedings of the First International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 74 of Proceedings of Machine Learning Research, PMLR, ECML-PKDD, Skopje, Macedonia, 2017, pp. 22–35. [36] P. Szymański, T. Kajdanowicz, A scikit-based Python environment for performing multi-label classification, ArXiv e-prints (2017). arXiv:1702.01460.