The paper is devoted to the problem of recognizing human actions in videos recorded in the optical range of wavelengths. An approach proposed in this paper consists in the detection of a moving person on a video sequence with the subsequent size normalization, generation of subsequences and dimensionality reduction using the principal component analysis technique. The classification of human actions is carried out using a support vector machine classifier. Experimental studies performed on the Weizmann dataset allowed us to determine the best values of the method parameters. The results showed that with a small number of action classes, high classification accuracy can be achieved.
Human action recognition is actively used in various fields: in creating human-machine interfaces, in entertainment, in ensuring public safety, etc.
Human actions recognition involves solving two problems [
2. Actually classification of the feature information obtained at the first stage.
To solve these problems, many approaches have been proposed, described in detail in [
To obtain the feature information, the authors of the paper [
To eliminate this problem, a generalizing approach related to the use of 3D motion history volume
(MHV) was proposed in [
Another approach to obtaining features is associated with the extraction of space-time interest
points (STIPs). Thus, the authors of [
In [
For classification of the obtained features, various approaches are used, namely, the support vector
machine (SVM) [
In this paper, an approach based on the dimensionality reduction using the principal component
analysis and subsequent classification using the support vector machine is used to solve the problem of
human action recognition. A similar approach was successfully used by us earlier [
The paper has the following structure. Section 2 describes the developed method for human action recognition. Section 3 describes the results of experimental studies performed on the Weizmann dataset. The conclusions and the list of literature is given at the end of the paper.
The method proposed previously [
At the first stage of the developed method, the moving person is detected in the video sequence. When
the video sequence source is a video surveillance camera, background subtraction methods are used
most frequently. The main idea of the methods of this class is to use a certain background model and
to decide whether the particular pixel belongs to the background or a moving object, based on its
correspondence to the background model. The background model is gradually refined over time.
Although the time-averaged observation image can be used as a background model in the simplest
applications, better results to this problem are given by more complex models, for example, [
In this paper, we use the background subtraction algorithm based on the mixture of Gaussian
distributions (Gaussian mixture model, GMM) [
We note that when choosing a method based on a mixture of Gaussian distributions, both our
preliminary experiments and the experience by other researchers in solving the problem under
consideration, were taken into account [
As a result of the first stage, the set of masks corresponding to individual frames of the video sequence is formed. Each mask reflects the result of the segmentation of a frame into the foreground area corresponding to a moving person and the background.
At the second stage of the method, obtained masks are processed as follows. First, the center of mass for each foreground region is calculated, then the linear sizes of the region are determined, and a framing (a truncation of the mask image) is performed. After that, the cropped image is resized (compressed) to the specified size.
Taking into account the time coordinate, the dimensionality of the sequence of masks, which describes the movement of a person, remains high even after the size normalization (framing and compression). In this regard, the fourth stage reduces the dimensionality of data describing the movement of a person.
For each sequence of frames containing motion, a set of subsequences of a given length is allocated.
Generation of subsequences is carried out with some specified step, starting from the beginning of the
original sequence. A detailed description of the allocation of subsequences is given in previous papers
[
For each selected subsequence, the vector of features is formed as follows: each normalized frame of the subsequence is expanded into a row, and the rows obtained for individual frames are concatenated to each other.
The feature vectors of the subsequences of all sequences form the input matrix for the dimensionality reduction stage.
Both linear and nonlinear methods are used to reduce the dimensionality of multidimensional data.
Linear methods such as principal component analysis (PCA) [
In this paper, we use the principal component analysis technique, as the most often used in similar
cases and in other tasks (for example, see our previous papers [
When principal components are found, the projection of feature vectors onto the first N principal components is taken as a feature description.
The features obtained as a result of the principal component analysis are used to train the classifier
Support Vector Machine (SVM) [
Note that the description given above is valid for the training mode in which the parameters of the dimensionality reduction and classifier are configured. In the testing mode, the data is processed in the same way, except that the parameters of the linear transformation are fixed to the values obtained in the training mode, and the trained SVM classifier performs the classification.
The proposed method was implemented in C ++ using the OpenCV library. A PC based on the Intel Core i5-3470 CPU 3.2 GHz was used to perform experimental studies.
For the experimental study of the proposed method, the video sequences from the open Weizmann dataset (Figure 1) were used. This dataset contains sequences of binary images corresponding to individual frames of the video sequence, on which moving objects have already been extracted (foreground and background segmentation). The dataset contains video sequences for 9 people performing 10 different actions. The total dataset contains 90 sequences. Thus, there were 9 sequences in each class. The minimum sequence length was 28 frames.
The sequences were divided into the training and test sets containing four and five sequences correspondingly for each class. The sequences were pre-processed using the algorithm described in Section 2.2. Further, subsequences were generated according to Section 2.3. Then, the dimensionality reduction using the method described in Section 2.4 and classification using the algorithm described in section 2.5 were performed. To estimate the quality of the method, we used the classification accuracy, defined as the proportion of correctly classified objects.
In the first experiment, we studied the dependence of the classification accuracy on the step size used in the generation of subsequences. The length of the subsequence was 28 frames. The output dimension of the feature vectors formed in step 2.4 of the considered method was equal to 8.
The experimental results are shown in figure 2. It was experimentally determined that the best classification accuracy is achieved with small step values. In further experiments, a step equal to 2 was used.
100 % , cy 95 a r ccu 90 a ino 85 t a c iif 80 s s a lC 75 100,00 % , cy 95,00 a r u cc 90,00 a n ito 85,00 a c i ifs 80,00 s a l C 75,00 1 2 3 4 6 8 16
28 5
In the second experiment, we investigated the dependence of the classification accuracy on the output dimensionality of the feature vectors, formed in step 2.4 of the proposed method. The output dimensionality varied from 4 to 512, while other parameters remained fixed. In particular, the step used in the allocation of subsequences was 2 frames.
The results of the experiment are shown in figure 3. As can be seen from the above results, the best values of the classification accuracy are achieved for dimensionality 8, 16 and 64.
In the third experiment, we investigated the dependence of classification accuracy on the length of subsequences. The length of the subsequence varied from 4 to 28 frames, while other parameters remained fixed. Thus, the step used in the allocation of subsequences was 2 frames; the output dimension was 64. The results of the experiment are shown in figure 4. As it can be seen from the results, the best values of the classification accuracy are achieved with a subsequence of 28 frames.
As can be seen from the results of these experiments, with a relatively small number of classes (10 classes), a high (not less than 95%) classification accuracy can be achieved.
The proposed method for human actions recognition consists in the detection of a moving person in a video sequence, normalization of the size, generation of subsequences, dimensionality reduction using the principal component analysis and classification using the support vector machine.
The experiments performed on the Weizmann dataset allowed us to determine the best values of the parameters of the developed method. It was shown that with a small number of classes (10 classes), the proposed method provides on this dataset a high (with a wide range of parameters - at least 95%, and using the best values - up to 98%) accuracy of classification.
In the future, it is planned to expand the list of algorithms used to form a feature description and the
list of classification methods. Another possible direction of further research is the detection of
abnormal behavior (see, for example, [
The work was partly funded by RFBR according to the research project 17-29-03190 in parts of «1. Introduction» – «2. Methods» and by the Russian Federation Ministry of Science and Higher Education within a state contract with the "Crystallography and Photonics" Research Center of the RAS under agreement 007-ГЗ/Ч3363/26 in part of «3. Experiments».