Establishing a strong foundation for similarity-based return is a top priority in Case-Based Reasoning (CBR) systems. Deep Metric Learners (DMLs) are a group of neural network architectures which learn to optimise case representations for similarity-based return by training upon multiple cases simultaneously to incorporate relationship knowledge. This is particularly important in the Human Activity Recognition (HAR) domain, where understanding similarity between cases supports aspects such as personalisation and open-ended HAR. In this paper, we perform a short review of three DMLs and compare their performance across three HAR datasets. Our findings support research which indicates DMLs are valuable to improve similarity-based return and indicate that considering more cases simultaneously offers better performance.
Establishing a strong foundation for similarity-based return is a top priority in
CaseBased Reasoning (CBR) systems. Without a firm understanding of the similarity
between cases, CBR systems are poorly placed to offer solutions from previous
knowledge and become increasingly reliant on their adaptation component [
Deep Metric Learners (DMLs) are a group of neural network architectures which
learn to optimise case representations for similarity-based return. This is achieved by
training on multiple cases simultaneously. Early examples of DMLs trained upon pairs
of input cases [
Despite their growing impact on this domain, there remains a lack of literature examining the quality of representations obtained by different DML architectures for a HAR task. With this in mind, we are motivated to do a review of the performance of three DMLs (Siamese Neural Network, Triplet Network and Matching Network) across three HAR datasets (SelfBACK, PAMAP2 and MEx). The contributions of this paper are therefore as follows: (1) we offer an introductory review of three DML architectures and (2) present a comparison of their performance on three HAR datasets.
The paper is structured in the following manner. In section 2 we provide an introductory review of each of the DML architectures that we consider in this paper. In section 3 we detail the experimental setup of our comparative evaluation and give details of the three HAR tasks, while in section 4 we analyse and discuss the results. Finally in section 5 we present some conclusions. 2
In this section we explore the unique traits of several DML architectures; Siamese Neural Networks, Triplet Networks and Matching Networks. Though individual architectures possess distinct nuances, there are several themes which are consistent across DMLs. With this in mind, let us introduce some general notation used throughout this paper. Let X be a set of labelled cases, such that for x 2 X , the function y(x) returns the class label, y, of case x. In the context of this paper, we will define matching cases as those which have the same class while non-matching cases will have differing classes. The embedding function ✓ is an appropriate parameterisation of any function used to create the vectorised representation of a given x, while the function DW represents an arbitrary metric function to measure the distance between two vector representations.
Siamese Neural Networks (SNN) are deep metric learners which receive pairs of cases as input. The SNN architecture consists of two identical embedding functions, enabling the SNN to generate a multi-dimensional embedding for each member of a pair. Input pairs are labelled as either matching or non-matching respectively. Correspondingly, the objective of training is to minimise the distance between the generated embeddings for matching pair members while maximising the distance between embeddings for nonmatching pair members. Thus the overall goal of the network is the development of a space optimised for similarity-based return.
To achieve this goal, each training pair, p 2 P, consists of two cases from the training set, p = (xˆ, x¨). Whether the pair is matching or non-matching is governed by the relationship of the pivot case, xˆ, to the passive case, x¨. In the context of this work, the pair’s relationship class is established by comparing class labels of its members (1) (2) (3) (e.g. y(xˆ) with y(x0)). For this we use function Y (p), which returns p’s relationship class label, such that Y (p) = 0 to symbolise a matching pair when y(xˆ) = y(x¨), and Y (p) = 1 to symbolise a non-matching pair when y(xˆ) 6= y(x¨).
After input to a network, the generated embeddings for each member of a pair can be compared using a distance metric, DW (✓ (xˆ), ✓ (x¨)). This distance metric plays a key role in the unique contrastive loss used by SNN (as in Equation 3); it penalises members of matching pairs until they occupy the exact same spot in the space (using LG in Equation 1) and penalises members of non-matching pairs until they exist at least a set margin distance of ↵ apart (using LI in Equation 2) to calculate error for model update (see Figure 1). The error is then backpropagated over both embedding functions to ensure they remain identical.
LG = (1
YA) · DW (✓ (xˆ), ✓ (x¨)) 2 LI = YA · (max(0, ↵ L = LG + LI
DW (✓ (xˆ), ✓ (x¨))))2 Using both errors means the similarity metric can be directly learned by the network through the comparison of the actual pair label YA (which, as above, is equal to 0 for matching and 1 for non-matching pairs respectively) and the distance between pair members, while using the generated embeddings of pair members during distance comparisons ensures iterative model refinement. It is also these learned embeddings which act as the improved representation for similarity-based return after training is complete.
Designed for matching tasks such as signature verification [
Triplet networks (TN) are DMLs which learn from three input cases simultaneously. Together described as a triplet, these inputs are the anchor case (xa), a positive case (x+) and a negative case (x ). The anchor case acts as a point of comparison, meaning that the positive and negative cases are dictated by their relationship to the anchor (i.e.
Fig. 2: Triplet Network Architecture matching and not matching respectively). Similar to SNN, the objective during training is to minimise the distance between an anchor and its associated positive case and maximise the distance between an anchor and its associated negative case. However, considering three cases at once ensures that update of weights is more focused. This is because SNN are learning based on only one aspect at any given time (e.g. either pair members are alike, or not), meaning that more pairs are required to build the full picture. Considering three cases at once allows the triplet network to better understand the context of the anchor case.
A triplet network is comprised of three identical embedding functions, each of which creates an embedding for one input (see Figure 2) before the error is calculated using triplet loss:
L = DW (✓ (xa), ✓ (x+))
DW (✓ (xa), ✓ (x )) + ↵ (4) Like contrastive loss (see Equation 3), triplet loss is a distance based function. The formula will generate a loss value in situations where the distance between the anchor case and the negative case, DW (✓ (xa), ✓ (x )), is less than the distance between the anchor case and the positive case, DW (✓ (xa), ✓ (x+)). The network is therefore penalised until matching cases are closer than non-matching cases. A minimum boundary between non-matching cases is enforced by the margin ↵ .
Unlike SNNs, TNs were designed to be supported by a similarity-based return
component [
Matching Networks (MNs) [
n; where k is the number of representatives per cluster and n is the number of clusters in the dataset. The similarity between the query case and the support set case pairs (x0, x0i) are calculated with a suitable similarity metric (DW (x0, x0i)), and an attention mechanism in the form of weighted majority vote estimates the class distribution (see Equation 6 and 7). This is enforced by the loss function categorical cross-entropy, which quantifies the difference between the estimated and actual distributions.
✓ (x) = x0 a(x0, x0i) =
eDW (x0,x0i)
P|s| eDW (x0,x0i) yˆ = |S|
X a(x0, x0i) ⇥ y
MN was first applied in the domain of one-shot and few-shot learning with image
recognition [
In this section, we offer details of our evaluation of three deep metric learners - SNN, TN and MN. We perform an empirical comparison of the representations gained from each network architecture to determine their effectiveness within the HAR domain. For both SNN and TN we use k-NN accuracy as a proxy for representation goodness, while for MN we use classification accuracy from the model’s final parametric comparison between query and support set to indicate the quality of representations. We evaluate this by performing a one-tail t-test to establish statistical significance at a confidence level of 95% on classification accuracy. 3.1
SelfBACK [
MEx (Multi-modal Exercise Dataset) 2 dataset is a Physiotherapy exercise dataset that contains data collected using a pressure mat, a depth camera and two accelerometers placed on the wrist and the thigh. Data was recorded for 7 exercises with 30 users. Each user performed one exercise for a maximum of 60 seconds. Seven exercises included in the dataset are Knee rolling, Bridging and Pelvic tilt, The Clam, Extension in lying, Prone Punches and Superman. The pressure mat, the depth camera and the accelerometers data record frequencies are 75Hz, 15Hz and 100Hz respectively. In this work we only work with the data from two accelerometers for the purpose of comparability with other datasets.
PAMAP2 [
The following pipeline was applied on all datasets as pre-processing and to form cases by progressively converting a raw signal to a Discrete Cosine Transformation (DCT) feature vector.
1Public dataset available at https://github.com/rgu-selfback/Datasets 2Public dataset available at https://data.mendeley.com/datasets/p89fwbzmkd/1 3Public dataset available at http://archive.ics.uci.edu/ml/datasets/pamap2+physical+activity+monitoring (a) Convolutional ✓
(b) Dense ✓ 1. Use a sliding window of 500 timestamps to segment the original raw sensor signal.
(no overlap for SelfBACK and PAMAP2, 2 second overlap for MEx) 2. Extract 3-dimensional (x, y, z) raw accelerometer data from each sensor. 3. Apply DCT and extract most significant 60 features from each dimension. 4. Concatenate all DCT feature vectors from each dimension of all sensors to form the final feature vector. Lengths of resulting feature vectors for SelfBACK, PAMAP2 and MEx are 360, 540 and 360 respectively. 3.3
We define a nearest-neighbour classification task for all three HAR datasets - SelfBACK, PAMAP2 and MEx. We use Leave-One-Person-Out (LOPO) validation with each dataset and perform 34, 8 and 30 experiments for SelfBACK, PAMAP2 and MEx respectively. We record the accuracy of each experiment and present mean value as the performance metric.
All feature embedding functions (Figure 4a and Figure 4b) used the ReLU
activation and were trained using the Adam optimiser [
Parameter Batch size Training epochs Output length - feature embedding function Number of neighbours for kNN ↵ Mini-batch size Samples per class Classes per set Training cases Each dataset was evaluated against the kNN algorithm which sets the baseline for DML algorithms. Table 2 presents the comparative results we obtained with algorithms detailed in the previous section. An asterisk indicates that the performance improvement is statistically significant at 95% confidence interval compared to the kNN baseline and we highlight the best performing algorithm with bold text.
It is evident that the representations learned with DML algorithms are better for similarity-based return than the raw representation. With a single exception, all representations learned with DMLs outperform the baseline. In the SelfBACK dataset, all DML algorithms significantly outperform kNN baseline with both MLP and CNN feature embedding functions. MN algorithm achieves the best performance and the MLP embedding function earns a 1.34% performance improvement over CNN embedding function. Similarly, results on the MEx dataset show that all DML algorithms significantly outperform the kNN baseline and the MN algorithm achieves the best performance. Unlike the SelfBACK results, CNN feature embedding function edge out MLP embedding function with a performance improvement of 1.30%. These results suggest that considering more cases simultaneously is advantageous to the algorithm. Furthermore, the choice of MLP or CNN to operate as an embedding function seems to be problem specific.
We observe a distinct difference on the PAMAP2 dataset where TN algorithm with CNN feature embedding function outperforms all DML algorithms and baseline. In addition, the MN algorithm performs the poorest and does not significantly outperform the baseline. This dataset is comparatively smaller compared to other two datasets which can be a contributing factor for the reduced performance with MN algorithm. It is interesting that the TN algorithm significantly outperform the baseline even with limited training data. We plan to explore this insight further in future work.
Overall the results are indicative that deep metric learning methods can learn an effective feature representation for similarity calculations. Most importantly, the imDataset
Network SelfBACK MEx PAMAP2 kNN SNN TN MN kNN SNN TN MN kNN SNN TN MN
Raw provements to performance were statistically significant in almost every circumstance. This is an important insight for when using similarity based algorithms where we previously relied on hand-crafted feature engineering. 5
In this paper we review three DML algorithms to learn feature representations and evaluate them against hand-crafted feature representations in a similarity based classification task. We select HAR as the classification task as similarity based classification holds special significance in improving performance by understanding personal nuances. Our results show that the feature representations learnt with DML algorithms significantly outperform hand-crafted feature representations in the selected domain. These results highlight the potential of DML algorithms to create effective feature representations efficiently, which is crucial in domains such as case-based reasoning. In future we plan to extend this review to other domains and compare further aspects of DMLs, such as the performance of different case mining approaches.