A CBR approach to Human Activity Recognition (HAR) uses the kNN algorithm to classify sensor data into different activity classes. Different feature representation approaches have been proposed for sensor data for the purpose of HAR. These include shallow features, which can either be hand-crafted from the time and frequency domains, or the coefficients of frequency transformations. Alternatively, deep features can be extracted using deep learning approaches. These different representation approaches have been compared in previous works without a consistent best approach being identified. In this paper, we explore the question of which representation approach is best for kNN. Accordingly, we compare 5 different feature representation approaches (ranging from shallow to deep) on accelerometer data collected from two body locations, wrist and thigh. Results show deep features to produce the best results for kNN, compared to both hand-crafted and frequency transform, by a margin of up to 6:5% on the wrist and over 2:2% on the thigh. In addition, kNN produces very good results with as little as a single epoch of training for the deep features.
Human activity recognition (HAR) is the computational discovery of human activity
from sensor data and is receiving increasing interest in the areas of health care and
fitness [
HAR is generally considered as a classification problem where a classifier is trained to identify user activity from sensor data. A CBR approach to this problem makes use of
Copyright © 2017 for this paper by its authors. Copying permitted for private and
academic purpose. In Proceedings of the ICCBR 2017 Workshops. Trondheim, Norway
a kNN classifier in order to facilitate similarity-based reasoning and explanation.
However, the effectiveness of a kNN classifier depends on the quality of the feature
representation used. Different feature representation approaches have been proposed for HAR,
from shallow hand-crafted features to frequency transform features e.g. Fast Fourier
Transforms (FFT) and Discrete Cosine Transforms (DCT) coefficients, and more
recently, deep learning approaches. All these approaches have had some degree of success
and setbacks in performance [
The main contribution of this work is an empirical evaluation of 5 different feature representation approaches across three different classes of features i.e. shallow handcrafted features, shallow frequency transformation features and deep CNN derived features, for kNN, using sensor data collected from two common body locations, the wrist and the thigh. Wrist data is more prone to random noise compared to data collected at other body locations (e.g. thigh) due to increased variations in movement and posture possible with the hand while undertaking activities. Our goal in this work is to understand which of these feature representations is better suited for the kNN classifier and to analyse any differences in feature performance that may exist between the wrist and thigh.
The rest of this paper is organised as follows: in Section 2, we highlight important related work on feature representation for HAR. Our dataset is described in Section 3. Evaluation is presented in Section 4 and conclusions in Section 5. 2
Many different feature extraction approaches have been proposed for accelerometer
data for the purpose of activity recognition [
This is the most common approach to HAR and involves the computation of a number of
defined measures on either the raw accelerometer data (time-domain) or the frequency
transformation of the data (frequency domain) [
Mean Dominant frequency Standard deviation Spectral centroid
Inter-quartile range Maximum
Lag-one-autocorrelation Mean Percentiles (10,25,50,75,90) Median
Peak-to-peak amplitude Standard deviation
Power Skewness
Kurtosis
Log-energy
Zero crossings
Root squared mean
Table 1. Hand-crafted features for both time and frequency domains.
While hand-crafted features have worked well for HAR, a significant disadvantage
is that they are domain specific. A different set of features need to be defined for each
different type of input data i.e. accelerometer, gyroscope, time-domain and frequency
domain. Hence, some understanding of the characteristics of the data is required. Also,
it is not always clear which features are likely to work best [
Frequency transform features extraction involves applying a single function on the raw accelerometer data to transform this into the frequency domain, where it is expected that distinctions between different activities are more emphasised. The main difference between frequency transform and hand-crafted features is that the coefficients of the transformation are directly used for feature representation without taking further measurements. Common transformations that have been applied include Fast Fourier Transforms (FFTs) and Discrete Cosine Transforms (DCTs).
FFT is an efficient algorithm optimised for computing the discrete Fourier transform of a digital input. Fourier transforms decompose an input signal into its constituent sine waves. In contrast, DCT, a similar algorithm to FFT, decomposes a given signal into it’s constituent cosine waves. Also, DCT returns an ordered sequence of coefficients such that the most significant information is concentrated at the lower indices of the sequence. This means that higher DCT coefficients can be discarded without losing information, making DCT better for compression.
For frequency transform feature extraction, a transformation function (DCT or FFT)
is applied to the time-series accelerometer vector !v of each axis. The output of is
a vector of coefficients which describe the sinusoidal wave forms that constitute the
original signal. Accordingly the transformed vector representations, x0 = (x), y0 =
(y) and z0 = (z), are obtained for each axis of a given instance. Additionally we
derive a further magnitude vector, m = fmi1; :::; milg of the accelerometer data for
each instance as a separate axis, where mij is defined as mij =
As with x0, y0 and z0, we also apply to m to obtain m0 = (m). The final feature
representation is obtained by concatenating the absolute values of the first l coefficients
of x0, y0, z0 and m0 to produce a single feature vector of length 4 l. The value l = 80
is used in this work, which is determined empirically. Further information on feature
representation using DCT and FFT can be found in [
Convolutional Neural Networks (CNNs) have been applied for feature extraction in
HAR, due to their ability to model local dependencies that may exist between adjacent
data points in the accelerometer data [
The input into the CNN in Figure 1 is a 3-dimensional matrix representation with
dimensions 1 28 3 representing the width, length and depth respectively. Tri-axial
acceleromter data typically have a width of 1, a length l and a depth of 3 representing the
x, y and z axes. A convolution operation is then applied by passing a convolution filter
over the input which exploits local relationships between adjacent data points. This
operation is defined by two parameters, D representing the number of convolution filters
to apply and C, the dimensions of each filter. For this example, D = 6 and C = 1 5.
The output of the convolution operation is a matrix with dimensions 1 24 6, these
dimensions being determined by the dimension of the input and the parameters of the
convolution operation applied. This output is then passed through a Pooling operation
which basically performs dimensionality reduction. The parameter P determines the
dimensions of the pooling operator which in this example is 1 2, which results in
a reduction of the width of its input by half. The output of the pooling layer can be
passed through additional Convolution and Pooling layers. The output of the final
Pooling layer is then flattened into a 1-dimensional representation and then fed into a fully
connected neural network. The entire network (including convolution layers) is trained
through back propagation over a number of generations until some convergence criteria
is reached. Detailed description of CNNs can be obtained in [
Note that once the CNN is fully trained, it can used to provide feature representations for use with other types of classifiers e.g. kNN. This is achieved by cutting off the trained network after the final pooling layer and just before the fully-connected neural network. Each training example is then passed through the convolutional network in order to obtain an abstract representation which is used to train the kNN classifier. A similar operation is performed for each test example to obtain an abstract representation which is passed to kNN for classification. 3
A group of 34 volunteer participants was used for data collection. The age range of participants is 18 - 54 years and the gender distribution is 52% Female and 48% Male. Data collection concentrated on the activities provided in Table 2.
Activity Description Walking Walking at normal pace Jogging Jogging on a treadmill at moderate speed Up Stairs Walking up 4 - 6 flights of stairs Down Stairs Walking down 4 - 6 a flights of stairs Standing Standing relatively still Sitting Sitting still with hands on desk or thighs
Table 2. Details of activities classes in our dataset.
This set of activities was chosen because it represents the range of normal daily activities typically performed by most people. Data was collected using the Axivity Ax3 tri-axial accelerometer 2 at a sampling rate of 100Hz. Accelerometers were mounted
on the right-hand wrists and right thighs of the participants. Activities are evenly distributed between classes as participants were asked to do each activity for the same period of time (3 minutes). 4
Evaluations are conducted using a leave-one-person-out methodology where each user’s data is held out for testing in turns, while the remaining 33 are used for training. In this way, we are testing the general applicability of the system to users whose data is not included in the trained model. Performance is reported using macro-averaged F1 and kNN is used for classification with euclidean distance and the parameter k = 5. The representations included in our comparison are as below: - Time: Time domain hand-crafted features - Freq: Frequency domain hand-crafted features - DCT: DCT frequency features - FFT: FFT frequency features - CNN: CNN deep features with soft-max classifier - CNN-kNN: CNN deep features with kNN classifier
For the CNN, after experimenting with different parameter settings, the final configuration used for thigh data had 3 convolution layers with 150, 100 and 80 convolution filters respectively. The configuration used for wrist data had 5 convolution layers with the same numbers of convolution filters as the thigh data in the first 3 layers and 60 and 40 convolution filters in the fourth and fifth layers respectively. Each convolution layer was followed by a max pooling layer. A convolution filter of size 10 and pooling size of 2 were used on all convolution and pooling layers respectively. The last pooling layer is connected to a fully connected network with 2 hidden layers, where the first layer had 900 units and second layer had 200 units. A dropout probability of 0.5 was used for each hidden layer. The final output layer had 6 units representing the 6 activity classes in our dataset and uses soft-max regression. Loss is computed using cross-entropy and the network is trained using back-propagation for 200 epochs. 4.1
Results of our comparative evaluation are shown in Figure 2. The best results for both thigh and wrist are achieved using deep features (CNN and CNN-kNN). This highlights the fact that kNN, using deep features, can rival the performance of state-of-the-art deep learners, while still providing the ability for similarity-based reasoning and explainability that makes kNN desirable. In general, HAR performance is higher using thigh data compared to wrist by a margin of up to 14.7% (for DCT). This indicates that the thigh is a much better position for HAR compared to the wrist. However, the benefit from deep feature representations is consistent on both wrist and thigh.
Out of the shallow features (Time, Freq, DCT and FFT), the best performance is
achieved using DCT. This is consistent with our previous findings [
Note that CNN-kNN outperforms CNN on both wrist and thigh at all 5 training epochs. Also, the performance of CNN-kNN is good (on par with Freq) even after a single epoch of training. These results are an important finding of this work and demonstrate the robustness of kNN in effectively using deep features, irrespective of the amount of time spent on training.
Finally, we analyse the effect of the depth of our network on the quality of deep features we are able to extract for kNN. Figure 4 shows the performance of CNN-kNN with different numbers of convolution layers between 3 and 5. Note that the best performance for the thigh is achieved using 3 convolution layers (0.949) and performance gradually decreases with the addition of more convolution layer (0.947 for 4 and 0.937 for 5). In contrast, performance on the wrist produces a significant increase (at 95% using a paired t-test) with additional layers from 0.73 for 3 layers to 0.84 for 5 layers. This indicates that deeper layers are required for effective feature extraction on more difficult datasets. However, a relatively shallow architecture seems sufficient for easier datasets. In this paper, we have presented an analysis of different feature representation approaches for the purpose of human activity recognition using kNN. These feature representation approaches can be broadly categorised into three classes: handcrafted, frequency transform and deep features. Evaluation is conducted using accelerometer data collection from two different body locations: wrist and thigh. Results show deep features to significantly out-perform the other representation types on both wrist and thigh by a margin of over 6:5% on the wrist, and 2:2% on the thigh. In addition, our evaluation shows kNN to be very effective at using deep features, even when a minimum amount of time spent in training these deep features.
Future work will investigate the use of RNN for feature extraction due to their ability to model the sequential relationship inherent in the time series accelerometer data.