The Siamese Neural Network (SNN) is a neural network architecture capable of learning similarity knowledge between cases in a case base by receiving pairs of cases and analysing the dierences between their features to map them to a multi-dimensional feature space. This paper demonstrates the development of a Convolutional Siamese Network (CSN) for the purpose of case similarity knowledge generation on the SelfBACK dataset. We also demonstrate a CSN is capable of performing classication on the SelfBACK dataset to an accuracy which is comparable with a standard Convolutional Neural Network.
Similarity knowledge is an essential component of an eective Case-Based Reasoning (CBR) system, but its generation can be a daunting task. Large complex datasets, where inter-feature relationships may exist, present a challenge to traditional similarity generation measures. Although similarity-based retrieval can oer numerous advantages during the retrieval phase, this can have a large initial cost. It is little wonder that recent research is targeting methods of harnessing deep learning methods to improve similarity knowledge generation.
A Siamese Neural Network (SNN) is a deep learning architecture which can learn similarity knowledge at a case-to-case level. SNNs have proven eective at learning similarity knowledge for a range of dierent domains including smartphone gesture classication and face verication [2; 4]. This paper presents the application of an SNN architecture to the SelfBACK 1 dataset2, which contains 1The SelfBACK project is funded by European Union’s H2020 research and innovation programme under grant agreement No. 689043. More details available: http://www.selfback.eu 2The SelfBACK dataset associated with this paper is publicly accessible from https://github.com/selfback/activity-recognition Copyright © 2017 for this paper by its authors. Copying permitted for private and academic purpose. In Proceedings of the ICCBR 2017 Workshops. Trondheim, Norway the accelerometer data for 34 users labelled with one of 6 activities. The main contribution of this paper is to demonstrate the successful application of an SNN as a means to develop similarity knowledge within a case base. In addition, this paper demonstrates that an SNN can perform a classication task on a level which is competitive with a typical Convolutional Neural Network (CNN).
This paper is organised into the following sections. Section 2 gives an overview of the research regarding learning similarity measure for use in CBR, as well as the SNN architecture and how it may be used as a means to develop similarity knowledge between cases which can be used for classication. Section 3 contains a description of the SelfBACK dataset and how it was used within the context of the presented research. Section 3 also describes our evaluation and details the setup of our experiments, including our classication method and network architecture, as well as pair creation method. Section 4 contains the results of our experiments and Section 5 highlights further work we aim to complete within this research area. 2 2.1
Learning eective similarity measures between cases can counter many of the
issues that plague the retrieval phase of CBR systems, such as retrieving suitable
results from extremely large and complex case bases, or retrieving results for
cases where some features cannot be explicitly described [
Knowledge-Intensive Similarity Measures (KISMs) have been shown to
improve retrieval in case bases where domain-specic knowledge is a key component
[
An SNN architecture consists of two neural networks that share identical weights and are joined at one or more layers. SNNs receive case pairs as input to both the training and testing phases to develop simlarity knowledge at an object-toobject level. An example architecture is shown in Figure 1. During the training phase, these pairs are labelled as either ‘genuine’ (if the examples share the same class) or ‘impostor’ (if the examples are of dierent classes). This allows the network to develop a multi-dimensional space based upon cases features, where ‘genuine’ pairs are pushed closer together and ‘impostor’ pairs are pulled further away from each other. The output of the identical neural networks (or ‘sub-networks’) are feature vectors for each member of the input pair. The distance between these vectors is measured at the similarity layer to ascertain whether they belong to the same class based upon a threshold.
SNNs use ‘contrastive loss’, which was introduced in [
The equations for contrastive loss are detailed in Equations (1), (2) and (3). YA and YP are binary values which are equal to 0 for genuine pairs and 1 for impostor pairs, where YA is the actual label, YP is the predicted label and M is the margin.
LG = (1
YA)YP 2 LI = YA(max(M
YP ; 0))2 L = LG + LI (1) (2) (3) 2.3
Initially made popular by [
Although introduced as a method of signature verication and binary
classication [
In [
We can observe from the range of examples above that SNNs are capable of learning similarity knowledge and performing classication upon a wide range of domains. However, there remain areas which require further exploration. In particular, literature regarding the structuring of pair creation is lacking, as is research which utilises SNNs within ensemble classiers. 3
The aims of this paper are two-fold; to show that an SNN could generate similarity knowledge within a case base, and to demonstrate the performance of an SNN as a method of human activity classication. To this end, we performed two experiments upon the SelfBACK dataset. 3.1
The SelfBack dataset consists of time series data collected from 34 users
performing dierent activities over a short period of time. Data was collected by
mounting a tri-axial accelerometer on the thigh and right-hand wrist of
participants at a sampling rate of 100Hz as they completed a script of set activities,
performing each for an average of three minutes [
Our experiments used the thigh dataset due to time limitations. Data was split into 5 second windows, meaning that there were between 160 and 180 cases per user and 1,500 features per case. This resulted in 6,084 cases of thigh data. These were then labelled as one of 6 activities (standing, upstairs, downstairs, walking, jogging, sitting) to create the full dataset. 3.2
Firstly, we implemented a Convolutional Siamese Network (CSN) upon the raw accelerometer thigh data, DCT thigh data and FFT thigh data. Pairs of cases were fed into the CSN and the convolutional sub-networks of the CSN learned to produce representative feature vectors of each case. We tested that the network had learned feature vectors which were representative of the original cases by measuring the euclidean distance between pair vectors at the similarity layer and comparing this to a threshold to identify whether a pair of examples belonged to the same class (a genuine pair) or to dierent classes (an impostor pair). If we identied that the distance between the genuine pair was less than a certain threshold (i.e. the space that should exist between cases of opposing classes) and the distance between the impostor pair was greater than this threshold, then we could reasonably assume that the case had been mapped to the correct space (or a very close approximation of it). We therefore used the percentage of correctly identied pairs as our accuracy metric.
Secondly, we implemented a CSN and CNN to perform classication on the raw thigh accelerometer data. The raw data was used in order to demonstrate a comparison between the two architectures which was unaected by preprocessing of the data. For this experiment, we completed a similar process to the previous experiment until the CSN had learned representative feature vectors for each case in the test set. Each feature vector from the test set was then compared with 6 randomly selected class representative vectors generated from cases in the training set. The distance between the unlabelled test vector and each class representative vector was measured, and the test case was identied as belonging to the same class as the nearest class representative vector. The accuracy of the experiment was the percentage of correctly classied cases. 3.3
A CSN was constructed from 2 sub-networks, which had 2 convolutional layers, a attening layer and 2 fully connected layers. The rst layers used tanh activation functions, while the nal layer used a softmax function. The network was optimised using Stochastic Gradient Descent and the hyperparameters in Table 1. The output of the sub-networks was a representative feature vector for each case member of the pair. Euclidean distance between these vectors could then be measured and compared with the threshold.
In order to be comparable with the CSN, the CNN created for comparison purposes was a close replica of one of the sub-networks outlined above. The only major dierence was the use of categorical cross-entropy for the loss function and backpropagating this loss with respect to the weights.
Implementations were run for10 epochs as the loss had reached a suciently low value by this point. Experiments were repeated 5 times and a mean percentage of accuracy calculated. There were many random elements at numerous stages of all implementations and so running for multiple iterations and taking the mean of the result accuracy was the only method to ensure that results were legitimately indicative of network performance. Data was split between train and test sets using leave-p-out cross-validation (LPOCV), meaning that the test set comprised of cases from p users, while all remaining users made up the training set. LPOCV was used due to the real-world constraints of the SelfBACK project, which involves being able to identify users’ activities based upon their similarity to other users. This oered an improvement in accuracy over randomly splitting the dataset, though the larger training set caused minor overtting. Experiments were completed with p set to 5 and 10, to test the eect that increasing test set size had on results.
At run time, the dataset is normalised using standard normal distribution, Equation (4), where is the mean of the training set and is the standard deviation of the training set. These values were taken from the training set because the test data represented a population of unknown size and distribution. x 2 X j x (4) 3.5
Pair creation in all experiments was completed after the data had been split into training and test sets to ensure that there was no cross contamination which could eect the results. For pair creation, we dened d as the number of cases in the full dataset and p as the number of cases to be left out for testing. The training set therefore contained n = d p cases.
Initially, we attempted to exhaustively create all pairs by matching every case with every other case. However, this resulted in the creation of P1d cases, which made pair formation extremely slow. Instead, two pairs were created for every case in the dataset; a genuine pair (with a random case of the same class) and an impostor pair (with a random case of a dierent class). This meant that every case was represented at least twice. We enforced equal genuine and impostor pair creation because generating truly random pairs led to an imbalance of more impostor pairs than genuine, at a ratio of approximately 5:1, and had a negative eect on classication. The number of training and test pairs were therefore 2n and 2p respectively. 4 4.1
The CSN was able to develop good similarity knowledge for all three time and frequency representations of the thigh dataset, though best results were obtained from the thigh DCT data. This is indicated by the percentage of correctly identied pair relationships, which is shown in Table 2.
Raw DCT FFT
Even the minimum result of 92.17% obtained on the raw thigh dataset using L10OCV demonstrates that more than 92% of test cases have been mapped to appropriate feature vectors. With this in mind, distance between these vectors can act as a proxy for similarity measurements at a case-to-case level. These results support the argument that cases of the same class are grouped closer together within the feature space and lend evidence to the idea that the CSN can be used to form the basis for similarity-based retrieval in a CBR system. 4.2
Comparing a CSN and CNN on the SelfBACK Dataset As a classier, the CSN did not perform as well as the CNN, although it acheived over 90% classication accuracy on both experiments. Although the CSN performed competitively on the L5OCV, the CNN displayed much higher accuracy on the L10OCV experiments, as shown in Table 3.
These results indicate that the CSN requires more training data to be able to classify cases than a typical CNN does. The low variance in results across iterations, and resistance to increasing test set size, by both architectures, supports the idea that they generalise well even to large test sets. Although the CSN did not perform as well at classifying cases as the CNN, we argue that the generation of similarity knowledge as a by-product of the classication process is a non-negligible contribution. The main benet of using the CSN over a traditional CNN implementation is that the output of the CSN produces feature vectors of the original case base which are good representations of how each case ts into the case base as a whole and allows direct, accurate distance measurements as a proxy for measuring similarity between cases.
On reection, there may be a couple of reasons that the CSN did not perform as strongly on the classication task as the CNN. A more structured method of classifying the output test feature vector, such as exhaustive k-nn sorting or informed class representative selection, could potentially oer better classication results and may be worth further study. It is a distinct possiblity that the class representative which was randomly selected for comparison with the test vector was a poor representative of the class, and that may have inuenced the classication of test cases. Exhaustively comparing the test case with all training cases, or using a method of selection to pick class representatives may improve classication accuracy. 5
We have demonstrated that a CSN is capable of learning similarity knowledge on the SelfBACK dataset. In addition, we have demonstrated that a CSN can use this similarity knowledge to perform human activity classication on the SelfBACK dataset and can perform competitively with a CNN on this task.
In future work we would like to further explore SNN’s capacity to develop
simlarity knowledge between cases in order to determine whether this could be
used in some manifestation to develop similarity knowledge between features.
Our end goal is to use this similarity knowledge in order to populate the
similarity arcs which exist between information entities in a Case Retrieval Network
(CRN) [