This paper presents a study carried out in a controlled environment that aims at understanding behavioural patterns in climbing activities. Multi-Scale Jensen-Shannon Neighbour Embedding [8], a recent advance in non linear dimension reduction, has been applied to recordings of movement sensors in order to help the visualization of coordination modes. Initial clustering results show a correlation with jerk, an indicator of fluency in climbing activities, but provides more details on behavioural patterns. This work was partially funded by ANR-13-JSH2-004 Dynamov project.
Detection and qualification of behavioural patterns in climbing activities play a key point in performance management. For example, the correct qualification of his/her inter-limb coordination may help the performer to step into more efficient behavioural patterns.
In fact, a larger repertoire of inter-limb coordination patterns would enable
rapid and efficient behavioural adaptation to environmental constraints (such as
size, shape of hold, distance between holds) because climbers can switch between
patterns [
Past studies on coordination focus on statistics of the relative phase of 2
articulations that are seen as 2 oscillators [
In our climbing data, structures are unknown and may appear on different
scales: climbers, holds, paths, climbing order, learning curve, . . . Nevertheless,
standard clustering or dimension reduction methods, such as Stochastic
Neighbour Embedding (SNE), are known to be good at structure preservation only
for a particular scale. Recently, Multi-Scale Jensen-Shannon NE [
The learning protocol consisted in four climbing sessions, separated by two days of rest. Participants were instructed to self-pace their ascent, with the following task-goal: explore the way to climb as fluently as possible, i.e., without falling down while minimizing pauses and saccades of the body displacement. Instructions were not made too specific to allow new coordination patterns to emerge during exploratory behaviour under the varying task constraints.
Each session consisted in ascending randomly three different routes with grade ranged 5b-5c in the French Rating Scale of Difficulty (F-RSD) (ranging from 1 to 9). Each path was identifiable by colour and was set on an artificial indoor climbing wall by three professional certified route setters who ensured that routes match intermediate climbing ability. The three routes had the same height (10.3m) and they included the same number of hand-holds (20), which were bolted to a flat vertical surface. The holds were located at the same place on the artificial wall; only the orientation of the hold was changed: (i) the horizontal-edge route was designed to allow horizontal hold grasping, (ii) the vertical-edge route was designed to allow vertical hold grasping, and (iii) the double-edge route was designed to allow both horizontal and vertical hold grasping.
Each edge could also be grasped by the left and/or the right hand. At the fourth session, the participants climbed a fourth path, which mixed the three previous routes.
Fourteen participants took voluntarily part to this study, with mean age 22.7 2.9 yr, mean height: 176 5 cm, mean weight: 64.2 5.8 kg. Seven individuals in this group have practised in indoor climbing wall, for three years, three hours per week and have skill level in rock climbing of grade 6a-6b in the F-RSD, which represents an intermediate level of performance. Seven other participants have only practised for 10h and have a skill level of climbing of grade 5b-5c, which corresponds to novice level of performance. The protocol was approved by the local University ethics committee and complies with the declaration of Helsinki. Procedures were explained to the climber, who then gave his written informed consent to participate. Here, participant names have been masqueraded. 2.2
Data collection and segmentation
The directions of the trunk and the limbs (3D unit vectors in Earth reference)
have been collected from small, wearable, inertial measurement units (IMU),
located on the hip, right and left wrists, right and left feet, with the North
magnetic as reference, and sampled at 100 Hz. Our IMUs corresponded to a
combination of a tri-axial accelerometer ( 8G), tri-axial gyroscope (1600 .s-1)
and a tri-axial magnetometer (MotionPod, Movea c , Grenoble, France). For
each climb, the fluency was assessed by the jerk coefficient [
The jerk can be seen here as a measure of smoothness of the hip trajectory and is used as an indicator of the expertise skills. For trajectory x : [O; T ] ! R3, the dimensionless jerk is defined as
Jx =
T 5
Z T d3x dt3 (s) 2
ds; ( x)2 0 where x is the length of the trajectory.
Based on the acceleration and the angular velocity of the 5 sensors attached
to the limbs and the hip, a first segmentation [
From this binary state segmentation of each limb, a global body state is determined directly using the concatenation of the previous segmentation, leading to 5 full-body states: Immobility (all sensors are immobile), Postural regulation (only the hip sensor is mobile), Hold Exploration (hip sensor is immobile, limbs sensors are mobile), Hold Change (last Hold Exploration before a traction), Traction (some limbs sensors and the hip senors are mobile). 3 3.1
In order to prepare dimension reduction for qualitative human interpretation and
for the pattern discovery with clustering, gyroscope, accelerometer and
magnetometer information are converted into a 3 3 rotation matrix that describes
each sensor in an Earth frame (North, West, vertical). The transformation is
performed through a complementary filter based algorithm described in [
From rotations to features
Rotation matrices belong to a compact manifold, the Lie group of rotations
so(3) [
The geodesic distance between rotations A and B is defined as the angle of the composition C of rotation B and the inverse of rotation A. If A and B are the same rotations then the angle of C is null. In the rotation group, the inverse of a matrix is simply its transpose. This gives, d(A; B) = arccos tr(A|:B) 2 1 :
A rotation geodesic mean M is defined as a rotation that minimizes the sum
of geodesic distances between itself and the studied set of rotations. It may not
be unique. The computation of M , a rotation mean, of n rotations Ri with
i 2 [1 : : : n] involved the following iteration process [
2. Project each Ri to the tangent space of the rotation manifold in Mt, 3. Compute the mean of Pi and project it back to the manifold, leading to the new mean estimation,
Pi = log (Mt|:Ri) ; Mt+1 = Mt: exp
n 1 X Pi n i ! ;
The log and exp operators are costly matricial operations in contrast to their
element-wise counterparts. Nevertheless, the log of a rotation matrix R can be
efficiently computed [
For the variance V of a rotation set, we choose the mean of the squared geodesic distances of each rotation to a rotation mean, namely,
n V = 1 X d(M; Ri)2 : n
i=1 As rotation mean M must minimize the geodesic distances, variance V is unique even if the rotation mean M is not.
Rotation signals are split into 20 sets corresponding to the 5 sensors and to the segmentation in 4 high-level states. For each of these sets, the rotation mean and variance are computed. Moreover, segmentation is sum-up by state count and state transitions probabilities. For each climb, it is summarized by a vector of 220 continuous features decomposed in 20 rotation mean matrices in R3 3, 20 rotation variances in R, a state count vector in R4 and a transition matrix in R4 4. Let us note that these features correspond to a hidden Markov model.
Further projection and clustering algorithms do not use features directly but rather distances between examples. Because of the manifold nature of our data, special care must be taken here. A rotation mean is a rotation itself, thus a Euclidean distance between the matrices of two rotation means is not suitable: one must use the geodesic distance on rotations defined above. Similarly, rotation variances are squared angles, thus one must use the geodesic distance on angles for this part of the features. That is why specific distances are computed separately on each component of the feature vector. Then the final distance is a weighted sum. 4
Dimension reduction through Multi-scale SNE
To facilitate visualization and preliminary qualitative interpretation, high
dimensional climbing features will be projected into a 2D space while trying to
preserve the data structure. In Stochastic Neighbour Embedding (SNE), a
nonlinear method proposed in [
The HD similarity ij between examples i and j is computed by, ij =
exp Pk;k6=i exp 2 ij 2 i(B)2
2 2 i(iBk)2 ; where ij is the distance between HD examples. It corresponds to the probability that j is the neighbour of i. Due to the bandwidth i(B) specific to ith example and perplexity B, similarities are not symmetric.
Perplexity B is a hyper-parameter that indicates the soft Gaussian number of neighbours to take into account around an example. Bandwidth i(B) corresponds to the soft radius around the ith example needed to encompass such a number of neighbours. Formally, in a preprocessing step called equalization, i(B) is chosen so that the entropy of the HD similarities of the ith example is equal to the log of perplexity B,
X ij log( ij )A :
0
j exp sij =
Pk;k6=i exp where dij is the distance between between ith and jth LD examples.
Since similarities are normalized like distribution, a Kullback–Leibler
divergence DKL can be used as a criterion to measure the adequation between HD
and LD similarities. In our case, we use the Jensen–Shannon divergence [
DJS ( jjs) =
DKL( jjz) + (1 )DKL(sjjz) ; where z = + (1 )s and is the mixing parameter.
In our data, structures may arise at different scales: the climber, the path, the order of the climbs. Nevertheless SNE methods are known to be good at preserving the data structure around the neighbourhood fixed by perplexity B. Global structures may be filtered out for a low perplexity. Small structure might be inaccurate for high perplexity.
Multi-scale JSE [
The full embedding process starts by computing a SVD decomposition of HD
features to initialize the LD projection. Then the chosen criterion is minimized
by a gradient descent on the LD projections. See [
Visualization of climbing data (b) By paths l A l A 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 201e2112 Pah2 0 1 2 3 4 5 61e12 t Pah3 0 1 2 3 4 5 61e13 t Pah4 0 1 2 3 4 5 61e11 t Pah t (a) Peggy 0 0 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 201e2111 Pah2 0 1 2 3 4 5 61e11 t Pah3 0 1 2 3 4 5 61e12 t Pah4 0 1 2 3 4 5 61e11 t Pah t l A l A 1 Pah2 0 t Pah3 0 t Pah t 1.0 0 h400..68 ta0.4 P0.2 0.00.0 Climber: Peggy
Climber: Eve Climber: Henry 1e12
Climber: Gerard
1e11 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 201e2112 Pah2 0 1 2 3 4 5 61e14 t Pah3 0 1 2 3 4 5 61e14 t Pah4 0 1 2 3 4 5 61e14 t Pah t
(b) Eve To get a more precise picture of behavioural patterns, we have applied a agglomerative clustering on the data. The agglomerative tree has been cut to get 6 clusters (number chosen by BIC).
The result of clustering is shown in figure 2 in a time-line for four selected climbers. Each graph is split into 4 parts: a time-line on all the climbs, and 3 time-lines for each climb route. The last fourth path is intended to test transfer learning in further experimentation and has been climbed only once. The background color indicates in which cluster a climb belongs, colour are consistent for the climbers. The rotation jerk has been plotted in white over the cluster time-line.
In figure 2(a), a learning effect can be seen: the two first climbs of each path display a different pattern than the following climbs. The jerk slowly decreases, which indicates better fluency over time, this is generally correlated with fewer explorations as the climber learns.
In figure 2(b), the same learning effect is seen, but the jerk makes a step at the change of patterns showing an exploration burst. Moreover on the last 4 climbs, the patterns of path 3 differs from path 1 and 2; showing a particular adaptation to this environment. That singular route effect does not appear in the jerk.
In figure 2(c), cluster changes occur later at climb 4 with a path effect that is also not seen on thr jerk.
In figure 2(d), the jerk makse a bump at the end of the training, showing difficulties for the climber to perform the task. As the first two climbers, a change of cluster occurs at climb 2 but then returns back to the original pattern. 6
The rotation features projected by MS-JSE and their clustering convey more information than the jerk on behavioural patterns: patterns are still correlated with the fluency indicator but show path effects that are not seen in the jerk because only a single value of jerk is computed for a climb. For example the timeline analysis of clusters highlight that climbers start from one set of patterns and evolved separately to different sets of patterns. Moreover, having both, jerk and clusters, indicates which pattern lead to better fluency (jerk decreases in parallel) and which patterns are typical to new exploration by the climber (jerk bumps and decreases). This is interesting for performers in order to qualify the behavioural patterns along the learning curve, because it would allow identifying efficient vs. non-efficient patterns, and shared patterns between climbers.
In the future, we will investigate MS-JSE and clustering on a state segmentation basis, in order to provide finer pattern grain than one climb.