Learning psychomotor skills is challenging because motion is fast, relies strongly on subconscious mechanisms, and instruction typically disrupts the activity. As such, learners would pro t from mechanisms that can react swiftly, raise subtle mistakes to the conscious level, and do not disrupt the activity. In this paper, we sketch a machine learning-supported approach to provide feedback in two example scenarios: running, and interacting with a robot. For the running case, we provide an evaluation how motions can be compared to highlight deviations between student and expert motion.
Teaching bene cial psychomotor skills { such as moving healthily or interacting
with a robot companion { is challenging [
In this paper, we sketch a machine-learning-supported approach for feedback for two psychomotor activity scenarios, namely running and interacting with a robot. Our approach is intended as an inspiration for a general template that can be applied across a wide range of psychomotor skills. For the case of running, we provide a rst analysis using dynamic movement primitives which shows that we can abstract from irrelevant variations in psychomotor data and hone in on deviations from expert demonstrations that may be indicative of mistakes. Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
To avoid injuries in running, a special wearable assistant was created, using
an electrical muscle stimulation (EMS) device and an insole with force sensing
resistors [
Training of new skills by means of wearable technologies and augmented
reality is supported by a conceptual reference framework, which enables capturing
the expert's performance and provides various transfer mechanisms [
The Visual Inspection Tool (VIT) facilitates annotation of multimodal data
as well as the processing and exploitation for learning purposes [
To describe running motion, we rely on dynamic movement primitives (DMPs)
[
x(t) + v(t) + f (t);
where x(t) models the location of a joint at time t, v(t) the velocity of the
joint at time t, 2 R+ is a hyperparameter determining the period length of
the system, 2 R+ and 2 R+ are hyperparameters determining how fast
the dampened spring system counteracts disturbances, and f (t) is the forcing
term which models the speci cs of our motion. In DMPs, this forcing term is
always a linear combination f (t) = PkK=1 k(t) wk= PkK=1 k(t) of (learned)
coe cients wk with nonlinear basis functions k. For this work, we use the
following rhythmically repeating basis functions as suggested in [
The main strengths of DMPs are that we can t the coe cients wk to data
via simple linear regression, and that we can replay a motion at arbitrary speed
(1)
(2)
(by adjusting ) for arbitrary long times (by executing the system in
Equation 1 for longer). DMPs have been particularly popular in robotics to mimic
human demonstrations [
The aim of our project is to support learners or trainees in developing speci c psychomotor skills by means of immersive learning environments. The expected solutions will combine AI approaches processing multimodal data from suitable sensors by machine learning techniques in order to analyze performance and detect faults, and nally automatically generating individual feedback.
We consider two application cases: running and collaboration with a robot. From the learning perspective, they are di erent. In running (the concrete aim apparently depends on the target group and the objective, which we currently set as healthy running for a wide public) the trainee repeats (relatively simple) rhythmic movements again and again, but the movements of various parts of the body should be in harmony and follow certain rules, in order not to harm the body and perform e ectively. It suggests a behavioristic approach of learning, when each error should be immediately reported to the person, in order to assign the message to the corresponding movement. So when the deviation in comparison with an optimal blueprint overcomes a threshold, a suitable feedback is given. For example, when the person does not lift the feet properly, an acoustic feedback is provided.
On the other hand, collaboration between a human and a robot consists of various actions on both sides, following a common aim. Here, the human actions are typically performed by hands (e.g. in an assembly process), but in this case the person needs to evaluate the current context and decide what to do next, e.g. whether a micro-aim has been achieved and one can proceed with the next step. Two types of skills are required { the ability to cooperate with the robot (e.g. learner nudges the robot on its empty arm) and the ability to ful ll the requested task (e.g. learner puts the lid on the box). This reminds cognitivistic learning approaches, where formative feedback (we distinguish corrective and reinforcing type) plays an important role, allowing trial and error. Therefore, more complex actions need to be assessed, usually considering whether a speci c micro-aim has been achieved. Nevertheless, immediate feedback cannot be excluded either, especially in case of dangerous operations.
What both application scenarios have in common is the necessity of a summative feedback, which evaluates a whole (training) unit or a (work) session. Here, di erent phases or sequences of actions can be analyzed, showing which parts were managed well and where is a potential for improvements.
The arti cial intelligence in our project essentially has the following tasks that are important for the learning process: { Modeling templates and movement patterns to guide learners: data sets are collected that represent expert performance in selected psychomotor processes. With this data, machine learning processes are trained for the use cases. { Detection of mistakes in the execution of movements of the learners compared to an optimal blueprint. Any deviation above a threshold triggers feedback. { Generating helpful feedback for learners: Detected errors must be processed in such a way that learners receive starting points for improving their processes, which they can cognitively process and implement psychomotorically.
joint angles DMP coe cients
Let us illustrate these tasks for the running example. We rst need to collect expert demonstrations that cover a wide range of reasonable and healthy styles of running. These demonstrations need to be recorded via motion capture devices to retrieve joint angle information that abstracts from the speci c body con guration. Further, we need to abstract from running tempo and phase shift.
During a learning episode, we record a learner's current running, convert
it into the same representation, and compare the latest cycle of the learner's
running to the most similar expert demonstration, resulting in a measure of
deviation. If the deviation exceeds a threshold, we provide auditive, tactile, or
visual feedback, e.g. coloring the deviating limb in a virtual avatar of the
runner [
We evaluate our proposed representation for the running case on a data set of 33
runs of three runners recorded in the Carnegie Mellon Motion Capture Lab1. Our
aim in this experiment is to abstract from particularities of a speci c run and
to identify the runner by comparing a run against other runs in the data base.
Each run is represented by Euler angles of the right and left femur as well as the
right and left tibia. We, then, derive a summary representation of each run by
training a rhythmic dynamic movement primitive (DMP). We used K = 12 basis
functions as this was su cient to achieve a local minimum in the reconstruction
error. As hyperparameters we chose ck = 2 Kk , h = log(0:1)=(cos(2 =K) 1),
= 32, and = =4, as recommended by [
Next, we evaluate the accuracy of identifying the runner in a leave-one-out crossvalidation across all runs. We use a -nearest neighbor classi er as implemented in sklearn2 with the number of neighbors varying from one to ve. To select nearest neighbors, we use the Euclidean distance on the DMP coe cients. As baselines, we also consider the Euclidean distance on the rst 128 frames of
2 https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.
KNeighborsClassifier.html
40 ] g ed 20 [ ed 0 t 20 u i l pm 40 a 0 50 frame 100 0 50 frame 100 the raw signal (because all signals where at least 128 frames long) and dynamic time warping as implemented in the edist package3. In this paper, we sketched an approach to support learning psychomotor skills using machine learning. In particular, we proposed a pipeline that compares a learner's activity to blueprints from experts and provides feedback whenever deviations are detected that exceed a threshold. To make this approach viable, our comparison needs to abstract from irrelevant factors such as body shape, tempo, or phase shifts. We evaluated dynamic movement primitives for the case of running and did achieve a representation that was abstract enough to identify the runner from running behavior. In future work, we wish to implement our pipeline fully for running and for a robot interaction scenario and evaluate its e ectiveness in supporting human learners.