Some psychomotor tasks require students to perform a specific sequence of poses and motions. A natural teaching scheme for such tasks is to contrast a student's execution to a teacher demonstration. However, this requires strategies to match the teacher demonstration of each motion to the student's attempts and to identify diferences between demonstration and attempt. In this paper, we investigate methods to automatically detect student attempts for poses with only a single correct teacher demonstration. We investigate relevance learning, prototype networks, and attention mechanisms to achieve a robust few-shot approach which generalizes across students. In an experiment with one teacher and 27 students performing a sequence of motion elements from the field of fitness and dance, we show that prototype networks combined with an attention mechanism perform best.s
Some psychomotor skills require us to execute a specific sequence of poses, such as in dance
choreographies, as well as during the repetitive execution of fitness moves, e.g. squats. While
research has emphasized the need for holistic teaching beyond mere imitation [
An additional challenge is posed by the scarcity of training data. For any specific pose,
we expect only a single (correct) demonstration by a teacher and only few annotated student
sequences. In other words, we are confronted with a few-shot learning setting, where we need
to share as much information across poses as possible to learn a viable model. In particular, we
draw upon prior work on prototype networks [
By contrast, metric learning is concerned with finding a distance function such that the
distance (, ) is low if and should be close and large if and should not be close [
Our contribution in this work is that we combine prototypical networks with an attention mechanism for the purpose of keypose recognition, where such techniques have not yet been applied, to the best of our knowledge.
Our task is to recognize times when a student tried to execute a certain pose . More specifically, we record the motion of a student with a Kinect camera, yielding a time series 1, 2, . . . , , where each is a 26 × 3 matrix, storing the 3D position of 26 joints of the human body at frame . We wish to compare the student’s frames to expert demonstrations 1, . . . , for each of the poses and recognize frames where the student attempted a pose. Our basic approach is to compute some distance (, ) between student frames and teacher demonstrations and recognize pose at time , whenever (, ) < for some threshold . However, the Euclidean distance on the raw 3D positions is not suitable because it would be disturbed by diferences in body size and orientation, as well as deviances in joints that are irrelevant for the specific pose (Fig. 1, a).
To account for body size and orientation diferences, we first translate the time series to angular space, in particular the azimuth and elevation of a joint compared to its parent joint in the human skeleton (Fig. 1, b). Further, we apply learnable weights , for each pose and each joint , resulting in the distance (, )2 = ∑︀2=61 , · ‖ , − ,‖2 (Fig. 1, c).
To learn the weights ,, we require training data where the correct matching between student and teacher is known. In particular, assume that we have one time series 1 , . . . , per student . For each of these time series, assume that experts provided a label sequence 1 ,, . . . , , for each pose , where , = 1 if pose should be recognized at time , , = 0 if pose should not be recognized at time , and , = − 1 if we consider it irrelevant whether pose is recognized at time or not. a) Euclidean distance b) angular distance c) weighted ang. distance d) embedding distance
yt,k = 1
Given this kind of training data, we learn the weights , by minimizing the contrastive loss ℓ = ∑︁ ∑︁ ∑︁ (, )2 + ∑︁ [︀ 1 − (, )2]︀ +, (1) =1 :,=1 :,=0 where [1 − ]+ = max{0, 1 − }. This loss punishes large distances (, ) if , = 1 and distances below 1 if
, = 0 (Fig. 2). In other words, this loss tries to ensure that we can
recognize pose correctly by checking if (, ) is smaller than = 1. We can optimize this
loss by standard, gradient-based non-linear techniques, such as L-BFGS. We call this approach
relevance learning, in line with [
Note that this scheme is, essentially, a simple metric learning scheme [
Unfortunately, relevance learning is limited to situations where the set of poses is fully known.
For every new pose , we need to record new training data to train new weights ,, which
may be infeasible. Instead, we would prefer an approach which can be applied to new poses
without any re-training. To that end, we apply a two-layer feedforward neural network which
receives the expert demonstration of a pose as input and maps it to weights , = ().
By applying a sigmoid nonlinearity, we ensure that the weights remain in the range [
In addition to the attention mechanism, we also test the efect of a refined motion
representation. In particular, we use a 1D convolution with kernel length 31 and 32 filters, followed
by a sigmoid nonlinearity and a linear layer which reduces the 32 filter dimensions to a
single number1. Finally, we apply another sigmoid nonlinearity and a linear layer from the 26
joints to latent dimensions which integrates information across joints. Overall, we obtain
an -dimensional representation () of each motion frame . We compare to the correct
demonstration () via the distance (, )2 = ∑︀=1 () · (︀ () − ()︀) 2 (Fig. 1,
d). We learn all neural network parameters by minimizing loss (1), but we add a regularization
term · ∑︀=− 11‖ () − (− 1)‖2 to ensure that the learned motion representation is smooth
over time. We call this approach prototype attention net because it integrates the concept of the
attention net with the representation approach of prototypical networks [
As a final model, we also consider a prototype net where we omit the attention net and set the weights () to 1, instead.
We compare relevance learning, attention net, prototype net, and prototype attention net on a dataset of 27 students and one teacher, all executing a series of 25 fitness and dance motion elements (namely squat, raise arms (2x), tree squat, “pirate” (6x), elbow to knee (3x), crossover (2x), airplane, airplane squat, standing, lunge (4x), T-pose, clock (2x)) while being recorded with a Kinect camera. We had 15 female and 12 male participants with mean age 27 (std. 4.49 years). Table 1 displays the self-reported prior experience of the participants with sports in general and video tutorials in particular.
We note that the annotators only annotated the first frame where the students attempted a certain pose. To arrive at complete labels ,, we used a heuristic scheme where we automatically labeled 30 frames after the actual annotation with , = 1 and extended the annotation further as long as the 3D marker positions did not change beyond 5 times the average distance between adjacent frames. We further set
, = − 1 for the 30 frames before and after the annotated 132 was chosen as the next power of 2 above the number of keyposes, which was 25. However, future work could investigate more hyperparameter combinations. region to ignore cases where the students was already/still close to the target pose but not close enough to count as training data.
Even after this preprocessing, though, our dataset is highly imbalanced: , = 1 is relatively rare, whereas
, = 0 is common. Accordingly, we do not report accuracy but recall, precision, and F1 score, as well as the area under the precision-recall curve (AUC).
Table 2 shows the results. As we can see, the prototype attention net performs best, according to all measures. A Wilcoxon signed-rank test revealed that the AUC for relevance learning and the attention net were both significantly lower ( < 10− 3) but the AUC of prototype net and prototype attention net was statistically indistinguishable. This finding indicates that representation learning is more crucial than joint weighting in achieving good keypose detection results, at least on this particular dataset.
We considered the problem of keypose detection for a sequence fitness and dance motion elements in a few-shot setting, where only a single teacher demonstration and few student demonstrations per pose exist. To detect keyposes, we evaluated methods which compute a distance between teacher demonstrations and student frames and detect a keypose if the distance is below 1. We compared several schemes to arrive at a distance, namely (1) relevance learning, which optimized joint weights for each keypose, (2) an attention neural net which inferred the joint weights from the respective teacher demonstration, (3) a prototype network which represented both teacher and student motion in a latent space before computing distance, and (4) a combination of prototype and attention network. As expected, the prototype attention net (4) performed best but we found that the prototype net (3) performed nearly as well. Therefore, we conclude that representation learning is more crucial compared to attention, at least for our example.
In future work, it should be investigated how well a prototype network generalizes to new keyposes it was not trained on and how far performance can be improved with refined architectures. Beyond keypose detection, future work should investigate the ability to recognize entire motion, in addition to purely static poses. For all these future research opportunities, we believe that our proposed loss function and training scheme can contribute to robust detection approaches, which in turn can become a crucial component for new feedback methods in psychomotor learning.
Funding by the German Federal Ministry for Research and Education (BMBF) for the project MILKI-PSY (grant no. 16DHB4014) is gratefully acknowledged.