A real-world solution for Human Activity Recognition (HAR) should cover a variety of activities. However training a model to cover each and every possible activity is not practical. Instead we need a solution that can adapt its learning to unseen activities; referred to as open-ended HAR. Recent advancements in deep learning have increasingly begun to focus on the need to learn from few examples, referred to as k-shot learning and to go beyond this to transfer that learning to situations with unseen classes, referred to as zero-shot learning. The latter is particularly relevant to our research in open-ended HAR; and as yet remains unexplored. This paper presents our preliminary work with Zero-shot Learning (ZSL) with a Matching Network to address openended HAR. A Matching Network has the desirable property of learning with few examples and so is well suited to explorations in ZSL. We evaluate Matching Networks for ZSL with a HAR dataset. We propose the use of a variable length support set at test time to overcome the search for the best support set combination that currently plagues the xed length support set size used by matching nets. Our results show that the variable length approach to be an e ective strategy to maintain accuracy whilst avoiding the combinatorial search for the best class combination to form the support set.
Activity monitoring with wearable sensors is a popular digital health
intervention incorporated in many health and well-being mobile applications. A study
conducted in 2015 concluded that 58% of smart phone users in US downloaded
health-care tness applications on their mobile, and 47% of those who
downloaded, stopped using them due to the high burden of data entry and loss of
interest [
An open-ended HAR solution should have the ability to recognise activities
without prior knowledge about them. Existing open-ended HAR methodologies
look at di erent unsupervised activity discovery techniques such as on-line
clustering [
Ability to incorporate personal traits when expanding beyond ambulatory
activities to exercises and sports improves performance and user acceptance.
Recent work in personalisation with Matching Networks has shown promising
results for personalised HAR [
This paper presents preliminary work on de ning ZSL for the HAR domain, reporting results with the SelfBACK dataset1. We expose the challenges of adopting ZSL with matching networks and propose the use of a variable length support set as a new test approach to perform e cient ZSL. A comparative study demonstrates the utility of this proposal.
The rest of this paper is organised as follows: in Section 2, we will formalise ZSL with Matching Networks in the domain of HAR, and in Section 3 we present a description of our dataset and experimental design. Results are presented in Section 4 followed by conclusions in Section 5. 2
Matching Networks can be viewed as an end-to-end neural implementation of
the otherwise static kNN algorithm. The network iteratively learns to match
a given target instance to a small set of instances called a support set [
More formally a matching network predicts label y^ for an example x^ guided by a support set S. Lets consider a dataset with set of X activity instances belonging to set of L activity classes. We de ne a support set S as in Equation 1. Cardinality of the support set is k ntr, where k is the number of instances per class. ntr is the number of classes in support set and ntr jLj.
S = f(x; y)jx 2 X ; y 2 Lg (1)
The matching net's training set, f(t1; S1); (t2; S2); : : : ; (tN ; SN )g, has N elements, where each target instance, ti, is a training instance pair, (x; y), and is never featured in its' support set, Si (i.e. ti 62 Si). A Matching Networks classi er model, , learns to recognise the activity class, y, for a given target instance, x, relative to support set, S.
At deployment, the user records a set of instances, X^, for a set of, L^, activity classes. We can view this as the user providing a small set of instances for model calibration. We predict the label, y^, for test activity instance, x^, relative to a support set, S^.
(x^; S^) ! y^
S^ = f(x; y)jx 2 (X [ X^); y 2 (L [ L^)g
(2)
Accordingly we de ne, S^, as in Equation 2. Cardinality of set S^ is k nte, where
nte is the number of classes in the support set at deployment. With one shot
learning [
We propose a deployment approach where the number of classes in the support set size is customisable. Accordingly we introduce condition, nte jLj+jL^j, which facilitates inclusion of all available classes in the support set. This allows the classi er to make an informed decision when predicting activity, y^, relative to the support set.
Figure 1 illustrates how the typical matching network with xed length support set are used for k-shot learning as well as ZSL. With ZSL we can see how the absence of the expected class in the support set can result in a poor classi cation outcome. One way round this is to try out several class combinations within the support set (potential for combinatorial explosion). The alternative is to expand the support size to cover as many as the expected number of classes at test / deployment time. 3
We compare two methodologies below to explore ZSL with Matching networks:
Test instance
t
e
S
tr
o
p
p
u
S
h
t
g
n
e
L
d
e
x
i
F
t
e
S
tr
o
p
p
u
S
h
t
g
n
e
L
d
e
x
i
F
t
e
S
tr
o
p
p
u
S
h
t
g
n
e
L
e
l
b
a
ir
a
V
{ the original xed length support sets where, nte = ntr; and
{ the proposed variable length support set where, nte jLj + jL^j
SelfBACK dataset was compiled with a tri-axial accelerometer data streams
belonging to 9 activity classes performed by 50 individuals. Accelerometer was
mounted on the right-hand wrist and each activity was performed for
approximately 3 minutes. Data was recorded in ftime-stamp, x, y, zg format at 100Hz
sampling rate. We use pre-processing steps used by [
We use the matching network architecture in [
For xed length support sets, we evaluated 81 possible combinations of test support sets when one class was considered as not part of training. A summary of results (minimum, maximum, average and median accuracies) from these combinations appear in Table 1 (columns under xed length support set). Minimum accuracy obtained with a test support set combination is always zero; this is when the support set combination does not contain the expected test class which at test time will be the calibration data. Note that random guessing of a class would achieve only 11% accuracy (as we have 9 classes). These results emphasise the importance of considering all possible classes in test support set. We present median of all combinations to show how average is a ected by having an experiment with zero accuracy. We observe some interesting interactions between the target instances actual class and class combinations in its support set. For instance with Ejogging, accuracy was heavily penalised when class downstairs is in the support set. Similarly behaviour was observed with class Eupstairs when walk-slow activity is included in the test support set. These results provide useful insights as to the e ect of di erent activities have on each other due to their inter-class similarities at an embedding level.
Results with ZSL using a Variable length support set is presented in the
last column of Table 1. We can observe that recognition of new activities (i.e.
not seen during training but with some calibration data at test time) has in
places signi cantly higher accuracy and surpass accuracies with the xed length
approach (e.g. Jogging and Sitting). With others; such as downstairs, walk fast
and upstairs yield accuracies under 50%. Looking at classi cation accuracies
from previous work on this data set [
We explore two problems that ZSL with Matching Networks pose: the problem of nding the best class combinations with a xed support set size at test time; and related to that the situation when the expected class may not be in the support set at test time. There is an optimum class combination that helps to classify unseen instances; however searching the space of possible combinations is a combinatorial problem. We resolve this by introducing a new test approach with variable length test support set. We experiment with a disjoint test set and yield comparatively better classi cation performance with variable length approach for several activities recognitions classes. In future we will look at how inter-class relationships in a support set might impact performance when applying ZSL. Additionally we will extend our evaluations to overlap test sets and other HAR data sets to further validate this approach.