Nowadays, smart living environments are equipped with various kinds of sensors which enable enhanced assisted living services. The availability of huge data volumes coming from heterogeneous sources, together with emerging of novel artificial intelligence methods for data processing and analysis, yields a wide range of actionable insights with the aim to help older adults to live independently with minimal supervision and/or support from others. In this scenario, there is a growing demand for technological solutions to monitor human activities and physiological parameters in order to early detect abnormal conditions and unusual behaviors. The aim of this study is to compare state-ofthe-art machine learning and deep learning approaches suitable for detecting early changes in human behavior. At this purpose, specific synthetic datasets are generated, which include activities of daily living, home locations and vital signs. The achieved results demonstrate the superiority of deep-learning techniques over traditional supervised/semi-supervised ones in terms of detection accuracy and lead-time of prediction.
Frail subjects, such as elderly or disabled people, may be at risk when their health
conditions are amenable to change, as it is quite common in case of chronic
conditions. That risk can be reduced by early detecting changes in behavioral and/or
physical state, through sensing and assisted living technologies, nowadays available in
smart-living environments. Such technologies, indeed, are able to collect huge
amounts of data by days, months, and even years, providing important information
useful for early detection of changes. Moreover, early change detection makes it
possible to alert formal/informal caregivers and health-care personnel in advance when
significant changes or anomalies are detected, before critical levels are reached and so
preventing chronic diseases. The huge amounts of heterogeneous data collected by
different devices require automated analysis; thus there is a growing interest in
automatic systems for detecting abnormal activities and behaviors in the context of smart
living and elderly monitoring [
The rest of this paper is organized as follows. The following Section 2 contains related works, some background and state-of-the-art in abnormal activity/behavior detection, with special attention paid to elderly monitoring through heterogeneous data collected with multi-sensor systems distributed over indoor environments. Section 3 describes materials and methods used in this study, and provides an overview of the system architecture, the long-term data generation, and the ML/DL techniques. The findings are presented and discussed in Section 4 and Section 5, respectively. Finally, Section 6 draws conclusions and some final remarks. 2
Today’s available sensing technologies enable long-term continuous monitoring of
activities of daily living (ADLs) and physiological parameters (e.g., heart rate,
respiration rate, blood pressure, etc.) in the home environment. Normally, both wearable
and ambient sensing are used, either alone or combined, as multi-sensor systems.
Wearable motion sensors incorporate low-cost accelerometers, gyroscopes and
compasses, whereas physiological parameter sensors are based on some kind of
skincontact biosensors (e.g., heart and respiration rates, blood pressure,
electrocardiography, etc.) [
On the other hand, ambient sensing devices are not intrusive in terms of body
obstruction, since they require the installation of sensors around the home environment,
such as cameras (monocular/stereo, time-of-flight, Lidar, etc.), microphones, sonars,
pyroelectric infrared (PIR) sensors, radar sensors, and pressure/vibration sensors.
Such solutions, blending into the home environment, are generally well-accepted by
end-users [
The learning setups for detecting/predicting behavioral changes can be categorized
into three main categories: supervised, semi-supervised, and unsupervised
approaches. In the supervised case, abnormalities (i.e., changes) are detected via binary
classification in which both normal and abnormal behaviors (i.e., activity sequences) are
labelled and used to learn a model [
For each category of learning setup, i.e., supervised, semi-supervised, and unsupervised, one ML-based and one DL-based technique are evaluated and compared in terms of detection performance and prediction lead-time at the varying of both normal behaviors (NB) and abnormal behaviors (AB). All investigated ML and DL techniques are summarized in Table 1. For that purpose, synthetic datasets are generated by referring to common ADLs and taking into account how older people perform such activities at their home environment (i.e., instructions and suggestions provided by geriatricians and leading researches were taken in careful consideration). The synthetic dataset includes six basic ADLs, four home locations in which these activities usually take place, and five levels of basic vital signs (i.e., heart and respiratory rates) associated with the execution of each ADL. The six ADLs are activity of eating (AE), housekeeping (AH), physical exercise (AP), resting (AR), sleeping (AS), toileting (AT). The considered home locations (LOCs) are bedroom (BR), kitchen (KI), living room (LR), toilet (TO). Regarding vital signs, for simplicity and without loss of generality, the only hear rate signal has been considered by dividing it into five levels (HRLs): very low (VL) < 50 beats/min, low (LO) ∈ [50–80] beats/min, medium (ME) ∈ [80–95] beats/min, high (HI) ∈ [95–110] beats/min, very high (VH) > 110 beats/min.
The objective of this study is to deeply evaluate the learning techniques reported in Table 1 by considering abnormal datasets, obtained by including the following perturbations: 1. Changing of the starting time of one ore more activities (St). This is a change in the starting time of an activity, e.g., having breakfast at 9 AM instead of 7 AM as usual. 2. Changing of the duration of one or more activities (Du). This change refers to the duration of an activity, e.g., resting for 3 hours in the afternoon, instead of 1 hour as usual. 3. Disappearing of one or more activities (Di). In this case, after the change, one activity is no more performed by the user, e.g., having physical exercises in the afternoon. 4. Swapping of two activities (Sw). After the change, two activities are per-formed in reverse order, e.g., resting and then housekeeping instead of housekeeping and resting. 5. Changing the location of one or more activities (Lo). One activity usually performed in a home location (e.g., having breakfast in the kitchen), after the change is performed in a different location (e.g., having breakfast in bed). 6. Changing in heartrate levels when performing one or more activities (Hr). This is a change in heartrate during an activity, e.g., changing from a low to a high heartrate during the resting activity in the afternoon.
Although, the sporadic presence of above mentioned changes is not enough to determine an abnormal condition, nonetheless a sustained change over days or months in activities, locations, and heartrate levels may be linked to an AB. Hence, the aim of this study is to evaluate, the ability of ML and DL techniques in predicting such sustained changes, with the objective to notify caregivers/doctors who can use historical sensor data to make decisions within the application domain of ambient assisted living.
In this study, both normal and abnormal long-term (1-year) datasets are realistically generated by using a probabilistic model based on Hidden Markov Model (HMM) and Gaussian process (GP). The evaluation metrics adopted in this study are sensitivity (SEN) and specificity (SPE), defined as follows: , =
+ where TP is the number of true positives, FP is the number of false positives, TN is the number of true negatives, and FN is the number of false negatives. The lead-time of prediction (LTP) is defined as follows: maximum number of days, before the day at which the change becomes stable, within which the future change can be predicted with the highest performance, i.e., maximizing TP and TN, and minimizing FP and FN. Thus, the higher the lead-time (in number of days), the better is the overall prediction performance. 3.1
In this study, since HMM is used for data generation (in contrast to other studies
where HMM is used for detection purposes [
The temporal dependency of activities generated from hidden states is handled by subdividing a day into four time intervals and modeling the activities in each time interval with a dedicated HMM sub-model. For each sub-model , thus, the first state being activated starts at a time modeled as a GP, while the other states within the same sub-model start in consecutive time slots whose durations are also modeled as GPs.
The ADL, LOC and HRL signals are, normally, sampled at different rates according to the specific variability during day time of each signal. For example, the minimum duration of ADLs is of about 10 min, so it is not useful to sample the ADL signal at 1 min interval. Nonetheless, a unique sampling rate can be adopted for all measurements. In this study, the sampling rate of 0.2 sample/min (i.e., 5 min interval between two samples) is selected for all signals. Each dataset is represented in matrix form with rows and columns equal to the total amount of observed days (i.e., 365 days) and to the total amount of samples per day (i.e., 288 samples), respectively. For each dataset, the matrix cells can take 120 different values obtained by combining 6 ADLs, 4 locations, and 5 HR levels. For example, the cell value AE-KI-ME indicates that the subject is eating her meal in the kitchen and her HR level is medium (i.e., between 80 and 95 beats/min). Finally, each 1-year dataset is represented by an image of 365 × 288 pixels with 120 levels of which an example is reported in Figure 2. Additionally, for the sake of understanding, each dataset can be represented by splitting it into three different images, referring to ADLs (6 levels), locations (4 levels), and HR (5 levels), as shown in Figure 3.
Furthermore, to assess the ability of ML and DL techniques (reported in Table 1) to detect behavioral abnormalities and changes, the model parameters (i.e., transition probabilities, emission probabilities, starting times, and durations) were randomly perturbed in order to generate various kind of abnormal datasets. Without loss of generality, each abnormal dataset included only one of the abovementioned changes (i.e., St, Du, Di, Sw, Lo, Hr) at a time or pairs of them, taken without repetitions (i.e., StDu, StDi, StSw, StLo, StHr, etc.).
In order to evaluate the detection performance of ML and DL techniques in Table 1, the HMM parameters (e.g., transition and emission probabilities, starting time and duration of activities, etc.) are gradually perturbed between the 90th and 180th day, by randomly interpolating the parameters of the normal and abnormal models. The resulting perturbed dataset consists of three parts: the first one, ranging from day 1 to day 90, is referred to normal behavior; the second one, from day 90 to 180, is characterized by gradual changes, becoming progressively more accentuated; the third one, starting from day 180, is very different from the initial normal period, the change rate is low or absent, and the subject’s behavior moves into another stability period. An abnormal dataset, referred to the St change type, is reported in Figure 4.
As already anticipated, the learning problem can be addressed by means of different learning setups, depending on the label availability. Correspondingly, there are three main detection approaches, i.e., supervised, semi-supervised and unsupervised, and which are taken into account in this study as discussed below.
Supervised detection is based on learning techniques (i.e., classifiers) requiring
fully labelled data for training. This means that both positive samples (i.e., ABs) and
negative samples (i.e., normal behaviors) must be observed and labelled during the
training phase. However, the two label classes are typically strongly unbalanced,
since abnormal events are extremely rare in contrast to normal patterns that instead
are abundant. As a consequence, not all classification techniques are equally effective
for this situation. In practice, some algorithms are not able to deal with unbalanced
data, whereas others are more suitable thanks to their high generalization capability,
such as support vector machine (SVM) [
In real-world applications, the supervised detection workflow described above is
not applicable due to the assumption of fully labelled data, on the basis of which
abnormalities are known in advance and correctly labeled. However, dealing with
elderly monitoring, abnormalities are not known in advance and obviously cannot be
purposely performed to train the detection algorithms. Semi-supervised detection is
usually achieved by introducing the concept of one-class classification, whose
state-ofthe-art implementations—as experimented in this study—are one-class SVM
(OCSVM) [
The most flexible workflow is that of unsupervised detection. It does not require that abnormalities are known in advance but, conversely, they can occur during the testing phase and are modelled as novelties with respect to normal (usual) observations. The main idea is that extracted features are scored solely on the basis of their intrinsic properties. In order to decide what is normal and not, unsupervised detection is based on appropriate metrics of either distance or density. The distance used in this study is defined as follows:
( ̅) = ∑ =1| | ln( ), (3) where = 288 and ̅ = ( 1, 2, … , ) ∈ ℝ is a day of observation, i.e., a row of the matrix dataset.
Clustering techniques can be applied in unsupervised detection. In particular,
Kmeans is one of the simpler unsupervised algorithms that address the clustering
problem by grouping data based on their similar features into K disjoint clusters. However,
K-means is affected by some shortcomings: (1) sensitivity to noise and outliers; (2)
initial cluster centroids (seeds) are unknown (randomly selected); and (3) there is no
criterion for determining the number of clusters. The Weighted K-Means [
For the experimental purpose, 31500 datasets were generated, i.e., 1500 random instances for each of the 21 abnormalities, obtained by considering the abnormalities (St, Du, Di, Sw, Lo, Hr), together with all pairs of these abnormalities taken without repetitions. Each dataset represented a 1-year data collection, as a matrix (image) of 365 rows (days) and 288 columns (samples lasting 5 min each), for a total amount of 105120 values (pixels) through 120 levels. The feature extraction process was carried Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). out by considering a 50%-overlapping sliding window lasting 25 days, then leading to a feature space of dimension D = 7200.
Each dataset was divided into three parts: Upper (1st–90th days), middle (90th– 180th days), and lower (180th–365th days) regions. The feature vectors (i.e., ACTLOC-HRL sequences) belonging to the upper regions were negative samples (i.e., normal behavior), whereas those belonging to the lower regions were positive ones (i.e., AB). The middle regions were, instead, considered as prediction regions, characterized by gradual changes becoming progressively more accentuated. The aim is to classify the feature vectors belonging to the middle regions in order to predict the likelihood of a future change which will become increasingly relevant and stable from the 180th day onwards (lower region).
In both supervised and semi-supervised settings, regarding the SVM classifier, a radial basis function (RBF) kernel was used. The kernel scale was automatically selected using a grid search combined with cross-validation on randomly subsampled training data. Regarding the CNN-based supervised detection, the network structure included eight layers: four convolutional layers with a kernel size of 3 × 3, two subsampling layers, and two fully connected layers. Finally, the two output units represented, via binary logical regression, the probability of normal and abnormal pattern behaviors.
The stacked auto-encoder (SAE) network was structured in four hidden layers, and the sliding-window feature vectors were given as input to the first layer, which thus included 7200 units. The second hidden layer was of 900 units, corresponding to a compression factor of 8 times. The following two hidden layers were of 180 and 60 units, respectively, with compression factors of 5 and 3 times. In supervised detection settings, the six abnormal datasets were joined in order to perform a 6-fold crossvalidation scheme. In semi-supervised detection settings, instead, only normal data from the same dataset were used for training, while testing was carried out using data from day 90 onwards.
Regarding the convolutional auto-encoder (CAE) structure, the encoder included
three convolutional layers with a kernel size of five, five, and three, respectively,
followed by a fully connected layer. The decoder structure was a mirror of the
encoder one. All experiments were performed on an Intel i7 3.5 GHz workstation with
16GB DDR3 and equipped with GPU NVidia Titan X using Keras [
The achieved results are reported in Table 2 in terms of specificity, sensitivity and LTP for all learning techniques evaluated in the present study. Note that the latter is related to SEN and SPE since LTP refers to the average number of days, before the 180th day (from which the changed behavior becomes stable), at which the change can be detected with such sensitivity and specificity. The longer the LTP the earlier the change can be predicted.
Generally, detecting ABs by using supervised methods presents the shortcoming of requiring both positive samples (i.e., changing activity sequences) and negative samples (i.e., habitual activity sequence) for model training. The SVM-based and CNNbased detection have been evaluated by training models with positive and negative samples taken from different datasets, in order to reproduce more accurately the reallife conditions mentioned above. Due to the lack of real data for training discussed above, the supervised approaches achieved the lowest detection performances.
The problem of training data availability is mitigated with semi-supervised techniques, since only negative samples (i.e., normal behaviors) are required, which are quite abundant in everyday activities. However, the main difficulty is to select training samples that are most representative of normal behaviors. The semi-supervised approaches evaluated in this study, i.e., OC-SVM and SAE, achieved intermediary detection performances, although with lower prediction performance (LTP) due to the difficulty selection of suitable (negative) samples for training.
The most promising results were obtained with the unsupervised learning methods,
i.e., K-means (KM) and DC, in which no labeled data were necessary, allowing the
easy adaptability to different environmental conditions as well as to users’ physical
characteristic and habits [
Classical ML methods, such as SVM and OC-SVM, have to deal with the problem of learning a probability distribution from a set of samples, which generally means to learn a probability density that maximizes the likelihood on given data. Conversely, such density does not always exist, as what happens when data lie on lowdimensional manifolds, which is the case of change types involving a narrow range of values (e.g., if the change regards only few heartrate levels), or when training and testing data come from a different probability distribution (e.g., as in the case of supervised learning). Under such a point of view, conversely, DL methods are more effective because they follow an alternative approach. Instead of attempting to estimate a density, which may not exist, they define a parametric function (deep network) able to generate samples closer to data samples taken from the original data distribution (by hyper-parameter tuning).
The contribution of this study is twofold. First, a common data model able to represent and process simultaneously ADLs, home locations and vital signs as image matrices is presented. Second, the performance of state-of-the-art ML-based and DLbased detection techniques have been evaluated by considering large datasets, synthetically generated, including both normal and abnormal behaviors. The achieved results are promising and show the superiority of DL-based techniques in dealing with huge datasets characterized by different kinds of data distribution. Future and ongoing activities are focused on the evaluation of ML/DL learning techniques in different domains, such as clinical decision support system and predictive maintenance.