In this paper, a new method for the eficient clustering of human activity data is proposed. Unlike the traditional human activity clustering approaches, our method relies on the skeletal data recorded with the help of motion capture (mocap) systems to achieve the goal. The proposed method is structured around the kernel-based robust covariance descriptor. By introducing a data re-framing technique that eficiently utilizes the temporal properties of the human activity data, we have alleviated the data redundancy and insuficiency issues associated with action sequences. The optimization model developed encompasses the combined benefits of low-rank representation and least square regression. The formulation is strengthened by incorporating the temporal dependency of the human activity sequences with the help of a temporal Laplacian regularizer. With the proposed algorithm, a representation matrix is learned from the raw data, which is then used to perform subspace clustering. Experiments conducted on multiple human activity datasets reveal the ability of the proposed method to achieve better clustering results compared to state-of-the-art counterparts.
Human activity recognition (HAR) from action sequences remains a challenging research
topic in computer vision due to its multifaceted applications [
The common approaches to HAR with mocap data involve supervised learning methods. Though they guarantee good results, such methods are often susceptible to missing sample issues. Further, the requirement of a huge clean dataset for initial learning of the system poses a serious bottleneck to supervised HAR approaches [8, 10]. This paves a foundation for the need of having unsupervised HAR strategies. Though in unsupervised methods the aforementioned challenges faced by supervised methods are alleviated, they encounter other limitations. The main challenge in unsupervised HAR is posed by the fact that the task needs to be performed in a robust and accurate manner without any prior knowledge about the data samples [11].
Recent studies showcase the ability of subspace clustering algorithms in dealing with highdimensional data clustering problems [12]. The key idea in subspace clustering approach is to identify multiple low-dimensional subspaces from which the data originates [13]. The subspaces so identified house a cluster of data. This approach is commonly termed as the Union of Subspaces (UoS) model [14]. The popularity of subspace clustering approach has increased with the introduction of sparse subspace clustering approach in which a sparsity constraint is imposed on the coeficients in order to learn a sparse representation of the raw data [ 14]. Low-rank representation learning (LRR) [15] is another technique used with subspace clustering wherein the global structure of the data is considered while learning the coeficients. In LRR based approaches, a given dictionary is utilized to learn a low-rank representation of the data samples. The clustering results obtained with such low-rank representations are usually better [16]. Least square regression (LSR) [17] based subspace clustering in which the grouping of data samples are performed with the help of Frobenius norm operator is another promising approach in subspace clustering. The aforementioned approaches when used independently will not be suitable for HAR as they do not consider the time series information associated with the skeletal data. As a solution to this problem we have developed an approach called the time series activity clustering (TSAC) which utilizes the combined advantages of LRR and LSR. We have incorporated a kernel-based robust covariance descriptor to extract features out of the raw input data with the aim of exploring the non-linearity present in the data. Further, the temporal Laplacian regularizer is employed to capture the temporal dependency among the data samples.
The following are the main contributions of this work: 1. An unsupervised optimization model with improved performance is formulated for the clustering of human activity data. To efectively utilize the temporal dependencies of human activity sequences, a temporal Laplacian regularizer is introduced in the proposed model. 2. We have blended the LRR and LSR based subspace clustering approaches to achieve better clustering results. A clean dictionary is learned along with the representation matrix in
Data re-framing Input action sequences
Kernelized covariance descriptors
Vectorization and stacking
Feature matrix
Representation matrix
Affinity matrix
Subspace clustering Data pre-processing and feature matrix generation Time series activity clustering
The rest of the work is outlined as follows. The proposed method, the problem formulation, and the solutions obtained are presented in Section 2. Section 3 explains the experimental validation done using the proposed method. Finally, conclusions are drawn in Section 4.
In this section, we give a detailed outline of the proposed TSAC approach. The workflow of the proposed approach is shown in Fig. 1.
A sequence of human action is a collection of action timestamps evolved over a period of time. Two important observations can be made with reference to a given collection of action sequences as mentioned below: 1. The number of timestamps corresponding to each of the action sequences may not necessarily be the same. This disparity often appears as a bottleneck in generalizing any algorithm dealing with human activity recognition. 2. As the number of timestamps in an action sequence increases, it introduces additional computation overhead. That will lead to a rise in demand for resource utilization.
The aforementioned challenges can be addressed by standardizing the number of timestamps for all the action sequences under consideration. If the number of timestamps is standardized to be ‘N ’, then data pruning methods need to be employed on action sequences having more number of timestamps and data augmentation needs to be performed on action sequences having lesser number of timestamps. The temporal smoothness property of human action sequences can be conveniently utilized to achieve this goal
Human action sequences are temporally highly correlated. As far as activity recognition is concerned, this correlation leads to redundant information. Often, the complete set of frames of a recorded action sequence is not essential to perform activity recognition. Thus, we introduce a pruning technique, termed succession pruning, in which the alternative frames of the action sequence are removed to eliminate redundancy while maintaining the temporal properties of the action sequence. That will drastically reduce the amount of data to be processed and will also result in reduced computation overhead and resource utilization. Whereas in action sequences experiencing insuficiency of timestamps, we perform timestamp augmentation. In this process, the trailing end of the action sequence gets augmented with the terminal timestamps of the same action sequence. That is in line with the temporal smoothness property of the human action data.
Human actions are represented in the form of skeletal structures with the help of modern motion capture systems. Each timestamp of an action sequence can be represented as a collection of ‘n’ joints. Thus, the time stamp ‘t’ of an action sequence can be represented as e() ∈ R3× of 3D positions {e1(), . . . , e()}. The 3D coordinates of the iℎ joint of the skeletal structure corresponding to the tℎ timestamp can be denoted as, e() ∈ [(), (), ()]⊤.
Once the data re-framing process is completed, the raw action sequences with fixed number of timestamps are represented in the form of a feature matrix. It involves a two step process. In the first step, we use the concept of covariance to obtain the covariance descriptor of each action sequence. The use of covariance will help us to capture the changes pertaining to each joint of the skeletal structure [18]. If represents the temporal average of the timestamps of an action sequence, then the corresponding action sequence can be represented in the form of a covariance matrix as shown below.
Ψ = 1
∑︁[e() − ][e() − ]⊤ − 1 =1 (1) This process is repeated for each action sequence, resulting in a unique covariance descriptor for each input sequence.
Although covariance descriptors finds application in multiple domains, they cannot capture non-linearity present in the data. For making the feature matrix robust, diferent approaches are adopted to incorporate additional statistical information along with the covariance descriptor. This includes the entropy based approaches, the mutual information based approaches, and the kernel-based approaches. Among others, the kernel-based approaches have been used to simulate more complex models. The use of kernels improves the descriptive power of the covariance matrices. The work proposed by Cavazza et al. [18] showcases the benefits of using kernels in works related to human activity recognition. Motivated by this observation, we modify the expression given in Eq. (1) to incorporate the kernel function and obtain the following robust covariance descriptor. (2) (3) Ψ = 1
∑︁ [︀ (e()) − ]︀[ (e()) − ]︀ ⊤ − 1 =1 Here, (.) represents the kernel function and is the temporal average of the kernel entries. The choice of kernel function is application specific. We have used two kernel functions namely the polynomial kernel and the exponential kernel, out of which the later one have produced promising results.
The exponential kernel is defined as: (e()) = exp {︁
e() }︁ ( + )2 where > 0 and is the kernel bandwidth.
After obtaining the robust covariance descriptors for each action sequence, in the second step we generate the feature matrix. The covariance descriptor Ψ contains redundant information as it is symmetric along the main diagonal. In order to reduce the amount of data to be processed, we vectorize each covariance descriptor by retaining the upper triangular values alone. Later, we stack (as columns) each of the vectors so obtained to form the feature matrix X ∈ R× . Here, ‘’ is the length of the individual vectors and ‘k’ is the total number of action sequences.
Subspace clustering is performed by generating a representation matrix out of the feature matrix in state-of-the-art approaches. This is accomplished by using the self representation property of the feature matrix, resulting in the computation of a set of unique coeficients Y ∈ R× for the feature matrix X ∈ R× . This can be mathematically expressed as X = XY. The limitation of such an approach is that they tend to produce sub-optimal results if the sampling done is not suficient. Instead of using the self representation property, if we learn an eficient dictionary W ∈ R× , we can overcome the aforementioned problem. Given a dictionary W, the set of data samples X can be expressed as X ≈ WY. The set of coeficients Y can be eficiently obtained in an iterative manner by utilizing the underlying low-rank nature of Y. The low-rank property of Y can be capture with the help of LRR [15]. Since the rank minimization problem is inherently NP-hard, nuclear norm can be used as a substitute for rank minimization [17]. It is also important to obtain the intra-cluster correlation among the data samples to obtain better clustering results. To this end, we can use the principle of LSR [17]. By yielding the concepts of LRR and LSR we formulate an optimization problem as follows.
1 Ym,Win 2 ‖X −
WY‖F2 + 21 ‖Y‖F2 + 2‖Y‖* s.t .
WY‖F2 captures the reconstruction error, ||·|| represents the Frobenius norm operator, || · || * denotes the nuclear norm operator, and 1 and 2 are the balancing
Manifold regularization methods are eficient in incorporating the temporal dependency of data in the problem formulation [19]. Thus, by modifying the general Laplacian regularizer, which captures the spatial dependency of data, we have developed a temporal Laplacian regularizer ℒ(· ) as our interest is in the temporal information of the action sequences.
For a representation matrix Y, the temporal Laplacian regularization function can be defined
Z is a temporal Laplacian matrix, W̃︁ii = ∑︀=1 , and Z is a weight matrix (5) (6) (7) (8) (9) = {︃1 for | − | ≤ 2
0 otherwise where denotes empirically defined threshold value.
With the introduction of ℒ(Y), Eq. (4) is modified as, min Y,W 2
1
1 ‖X −
WY‖F2 +
2 ‖Y‖F2 + 2‖Y‖* + 3ℒ(Y) s.t . Y ≥ 0, W ≥ 0 The optimization problem given in Eq. (7) can be solved using the ADMM approach under the ALM framework. ADMM finds solution for the unconstrained optimization problem by splitting the problem into multiple sub-problems. As a first step, we will introduce three auxiliary variables E, F, and G to decouple the terms present in the formulation. This results in a formulation as shown below.
min Y,W,E,F,G 2 1 ‖X −
WY‖F2 +
21 ‖E‖F2 + 2‖F‖* + 3ℒ(G) The Augmented Lagrangian corresponding to Eq. (8) is given as:
L(E, F, G, Y, W) = 21 ‖X −
WY‖F2 +
21 ‖E‖F2 + 2‖F‖* + 3tr(GLTG⊤) + ⟨Φ1, Y − E⟩ + ⟨Φ2, Y − F⟩ + ⟨Φ3, Y −
G⟩ + 2 (‖Y − E‖F2 + ‖Y − F‖F2 + ‖Y − 2.4.3. Updating G:
F[+1] = argmin 2‖F‖* + ⟨2, Y −
F⟩ + 2 ‖Y −
F‖F2
The update expression for F is found using the singular value thresholding (SVT) operator as 2.4.1. Updating E: The update expression for E is obtained by solving the following sub-problem.
E[+1] = argmin 1‖E‖F2 + ⟨Φ1, Y −
E⟩ + 2 ‖Y −
E‖F2
By diferentiating Eq. (10) with respect to E and equating it to zero, the E update is given as, The update expression for G is obtained by solving the following sub-problem.
G G[+1] = argmin 3tr(GLTG⊤) + ⟨Φ3, Y − G⟩ + 2 ‖Y −
G‖F2 By diferentiating Eq. (14) with respect to G and equating it to zero, the G update is given as, G[+1] = (︀ Φ[3]
+ Y[])︀( 3(LT + LT⊤) + I)︀ − 1 2.4.4. Updating Y: By solving the following sub-problem, the update expression for Y can be obtained.
Y[+1] = argmin
Y 1 2 ‖X −
WY‖F2 + ⟨Φ1, Y −
E⟩ + ⟨Φ2, Y −
F⟩ + ⟨Φ3, Y −
G⟩ + (︀ 2 ‖Y −
E‖F2 + ‖Y −
F‖F2 + ‖Y −
G‖F2)︀ By equating the gradient of Eq. (16) to zero, the update expression for Y is given as, [︁
]︁− 1[︁ Y[+1] = (︀ W[])︀ ⊤W[] + 3 I ︀( W[])︀ ⊤X[] + (︀ E[+1] + F[+1] + G[+1])︀ − ︀( Φ[1] + Φ[2] + Φ[3])︀ ]︁ (10) (11) (12) (13) (14) (15) (16) (17) Solution of the above equation can be found as,
W[+1] = [︁X[](︀ Y[+1])︀ ⊤]︁[︁Y[+1](︀ Y[+1])︀ ⊤]︁− 1 Finally, the Lagrange multipliers are updated as follows: 2.4.5. Updating W: The W sub-problem is given as follows. (18) (19) (20) (21) (22) (23) (24) (25) W[+1] = argmin
W 12 ‖X −
WY‖F2 Φ[1+1] = Φ[1] + (︀ Y[+1] − E[+1])︀ Φ[2+1] = Φ[2] + (︀ Y[+1] − F[+1])︀ Φ[3+1] = Φ[3] + (︀ Y[+1] − FG[[] ]⃦⃦⃦ ⃦⃦ ∞∞∞ , ⃦⃦ E[+1] − , ⃦⃦ F[+1] − , ⃦⃦ G[+1] −
E[]⃦ FG[[]⃦⃦⃦]⃦⃦ ∞∞∞ ⎫ ⎬ < ⎭
The overall process involved in the proposed time series activity clustering algorithm is summarized in Algorithm 1.
Once the representation matrix Y is obtained, an afinity matrix Q ∈ R× is calculated. The accuracy of clustering is highly dependent on the afinity matrix Q. A usual approach in obtaining the afinity matrix is as shown below [14, 15].
But, the graph so constructed do not take into account the intrinsic relationships of the withincluster data points. But for data containing temporal information, the within-cluster data points are highly correlated. In order to take advantage of this information, an afinity matrix Q is calculated as follows. where, ‖.‖2 represents the ℓ2 norm operator.
Update Φ[2+1] with Eq. (21) Update Φ[3+1] with Eq. (22)
Update [+1] = [] 12: 13: 14: = + 1 15: Use (23) to check the convergence 16: end while
To verify the performance of the proposed algorithm, it was tested on multiple human activity datasets. The datasets considered include the Gaming 3D (G3D) [22], Florence 3D (F3D) [23], UTKinect-Action 3D (UTK) [24], MSRC-Kinect12 (MSRC) [25], MSR Action 3D (MSRA) [26], and HDM14 [27] datasets. By means of observation, the parameters , 1, 2 and are set to 5.2, 0.03, 18, and 0.7 respectively. The program development was done on a system with Intel Core i7 processor and a RAM of 16 GB, operating on 64-bit windows operating system with a clock frequency of 2.90 GHz.
Experimental validation was done on the following methods.
Subspace clustering approaches based on self representation: In this method, clustering using state-of-the-art clustering approaches are performed on the generated afinity matrix. The clustering methods considered include Spectral Clustering (SC) [28], Orthogonal Matching Pursuit (OMP) [29], K-means (Km) [28], SSC [14], and Elastic Net Subspace Clustering (EnSC) [29]. The results obtained using SSC is found to be superior to that of the counterparts [14].
SSC approaches with Data Pruning: In this method, the input skeletal data is pruned with strategies including min Φ [29], Temporal SSC [29], Threshold Temporal SSC [29], and Percentage Temporal SSC [29]. Later, a feature matrix is generated from the pruned data sequences, followed by the application of SSC [14] approach. This method converges quickly. Among others, the percentage temporal SSC approach gives better results.
TSC-Cov: This is a method that we had developed in one of our previous works. In this method, data re-framing was not performed and the kernel-based features were not incorporated in the covariance descriptor. But, we had used a new clustering approach named TSC-Cov for subspace clustering.
The performance of the proposed algorithm was evaluated against the above mentioned methods. The metrics used for quantitative evaluation include the accuracy, the Normalized Mutual Information (NMI), and the Adjusted Rand Index (ARI). The results obtained are tabulated in Tables (1-3).
For qualitative evaluation, the afinity matrix and clustering results obtained using the proposed method and the state-of-the-art methods are presented. Although the experiments were conducted on all the six datasets mentioned earlier, for the purpose of illustration, the results obtained with the UTK dataset [24] is given in this paper. proposed method have much denser block diagonal structure. This is an indication of quality of the clustering process.
Fig. 3 shows the clustering results obtained on the UTK dataset. For visual analysis, each cluster is assigned a unique color. The figure shows a comparison between SSC [ 14], Percentage Temporal SSC [29], Threshold Temporal SSC [29], TSC-Cov, and the proposed method with reference to the true labels. From Fig. 3 we can observe that clustering results obtained with the proposed method are comparatively better than the other methods.
The paper proposes a new method for clustering of human activity sequences in an eficient way. The proposed method involves the extraction of features from raw input data with the help of a kernel-based robust covariance descriptor. The optimization model developed uses the combined advantage of LRR and LSR based subspace clustering approaches. The concept of temporal Laplacian regularized dictionary learning is introduced in order to learn an efective representation matrix from the extracted data features. With the help of ADMM approach, the solution for the optimization problem is obtained. Performance of the proposed approach is compared with that of the state-of-the-art approaches in terms of accuracy, NMI, and ARI. Experimental results validate superiority of the proposed method in obtaining better clustering results as compared to that of the counterparts. Motion capture data often sufers from corruptions in the recorded information. To address this problem, robust human activity clustering algorithms can be developed in the future. Also, mocap information can be combined with other modalities of human activity data to achieve improved clustering results in challenging scenarios. [8] S. Park, J. Park, M. Al-Masni, M. Al-Antari, M. Z. Uddin, T.-S. Kim, A depth camera-based human activity recognition via deep learning recurrent neural network for health and social care services, Procedia Computer Science 100 (2016) 78–84. [9] J. K. Aggarwal, L. Xia, Human activity recognition from 3d data: A review, Pattern
Recognition Letters 48 (2014) 70–80. [10] M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, Q. Tian, Symbiotic graph neural networks for 3d skeleton-based human action recognition and motion prediction, IEEE Tran. on PAMI (2021). [11] A. Bagnall, J. Lines, A. Bostrom, J. Large, E. Keogh, The great time series classification bake of: a review and experimental evaluation of recent algorithmic advances, Data Mining and Knowledge Discovery 31 (2017) 606–660. [12] L. Parsons, E. Haque, H. Liu, Subspace clustering for high dimensional data: a review,
Acm sigkdd explorations newsletter 6 (2004) 90–105. [13] R. Vidal, P. Favaro, Low rank subspace clustering (lrsc), Pattern Recognition Letters 43 (2014) 47–61. [14] E. Elhamifar, R. Vidal, Sparse subspace clustering: Algorithm, theory, and applications,
IEEE Tran. on PAMI 35 (2013) 2765–2781. [15] J. Xue, Y.-Q. Zhao, Y. Bu, W. Liao, J. C.-W. Chan, W. Philips, Spatial-spectral structured sparse low-rank representation for hyperspectral image super-resolution, IEEE Tran. on Image Processing 30 (2021) 3084–3097. [16] J. Francis, A. Johnson, B. Madathil, S. N. George, A joint sparse and correlation induced subspace clustering method for segmentation of natural images, in: 2020 IEEE 17th India Council Int. Conf. (INDICON), 2020, pp. 1–7. [17] Z. Wu, M. Yin, Y. Zhou, X. Fang, S. Xie, Robust spectral subspace clustering based on least square regression, Neural Processing Letters 48 (2018) 1359–1372. [18] J. Cavazza, A. Zunino, M. S. Biagio, V. Murino, Kernelized covariance for action recognition, in: 2016 23rd Int. Conf. on Pattern Recognition (ICPR), 2016, pp. 408–413. [19] Z. Zhang, K. Zhao, Low-rank matrix approximation with manifold regularization, IEEE
Tran. on PAMI 35 (2013) 1717–1729. [20] W. Liu, X. Ma, Y. Zhou, D. Tao, J. Cheng, -laplacian regularization for scene recognition,
IEEE Tran. on Cybernetics 49 (2019) 2927–2940. [21] G. Casalino, N. D. Buono, C. Mencar, Part-based data analysis with masked non-negative matrix factorization, in: Int. Conf. on Computational Science and Its Applications, Springer, 2014, pp. 440–454. [22] V. Bloom, V. Argyriou, D. Makris, Hierarchical transfer learning for online recognition of compound actions, Computer Vision and Image Understanding 144 (2015). [23] L. Seidenari, V. Varano, S. Berretti, A. Del Bimbo, P. Pala, Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses, in: 2013 IEEE Conf. on CVPR Workshops, 2013, pp. 479–485. [24] L. Xia, C.-C. Chen, J. K. Aggarwal, View invariant human action recognition using histograms of 3d joints, in: 2012 IEEE Computer Society Conf.on CVPR - Workshops, 2012, pp. 20–27. [25] S. Fothergill, H. M. M. , P. Kohli, S. Nowozin, Instructing people for training gestural interactive systems, in: CHI ’12 Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, ACM, 2012, pp. 1737–1746. [26] W. Li, Z. Zhang, Z. Liu, Action recognition based on a bag of 3d points, in: 2010 IEEE
Computer Society Conf. on CVPR - Workshops, 2010, pp. 9–14. [27] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, A. Weber, Documentation Mocap
Database HDM05, Technical Report CG-2007-2, Universität Bonn, 2007. [28] Y. Lee, S. Choi, Minimum entropy, k-means, spectral clustering, in: 2004 IEEE Int. Joint
Conf. on Neural Networks (IEEE Cat. No.04CH37541), volume 1, 2004, pp. 117–122. [29] G. Paoletti, J. Cavazza, C. Beyan, A. Del Bue, Subspace clustering for action recognition with covariance representations and temporal pruning, in: 2020 25th Int. Conf. on Pattern Recognition (ICPR), 2021, pp. 6035–6042.