Human Activity Recognition is becoming a popular field of research in the last two decades. Understanding human behavior in images gives useful information for a large number of computer vision problems and has many applications like scene recognition and pose estimation. There are various methods present for activity recognition; every technique has its advantages and disadvantages. Despite being a lot of research work, recognizing activity is still a complex and challenging task. In this work, we proposed an approach for human activity recognition and classification using a person's pose skeleton in images. This work is divided into two parts; a single person poses estimation and activity classification using pose. Pose Estimation consists of the recognition of 18 body key points and joints locations. We have used the OpenPose library for pose estimation work. And the activity classification task is performed by using multiple logistic regression. We have also shown a comparison between various other regression and classification algorithm's accuracy on our dataset. We have prepared our dataset, divided it into two parts, one is used to train the model, and another is used to validate our proposed model's performance.
The goal of a Human Activity Recognition (HAR) system is to predict the label of a person's action from an image or video. This interesting topic is inspired by many useful real-world applications, such as simulation, visual surveillance, understanding human behavior, etc. Action recognition through videos is a well-known and established research problem. In contrast, image-based action recognition is a comparably, less explored problem, but it has gained the community's attention in recent years. Because motion activities cannot be estimated through the still image, recognition of actions from images remains a tedious and challenging problem. It requires a lot of work as the methods that have been applied to video-based systems cannot be applicable in this. However, the approach is not the only problem faced in this task. There are many other challenges too, especially the changes in clothing and body shape that affect the appearance of the body parts, various illumination effects, estimation of the pose is difficult if the person is not facing the camera, definition, and diversity activities, etc.
Activity recognition through smartphones
and wearable sensors is very common; there are
various benchmarks available. But these
systems rely on collecting data from sensors
installed on the devices and user needs to wear
these devices that are uncomfortable in
practical. Vision-based systems are a better
alternative for this kind of problem due to the
fact that the user doesn't need to carry or wear
any device. Instead, tools like a camera are
installed in the surrounding environment to
capture data [
The rest of the paper is structured as in section 2 literature survey of some selected research papers in the area is mentioned. Section 3 contains the methodology and architecture of the proposed approach. A brief description of dataset and evaluation metrics (like precision, recall, and f1-score) used in this work is given in sections 4 and 5, respectively. Section 6 contains experiments and results of various classification algorithms applied in this work. Section 7 finally concludes the work with some future direction.
Research has recently begun to recognize
the behavior of humans from the images.
Compared to the video-based action
classification, the number of research papers
and journals are less. We have stated some
techniques used for HAR. Four types of
approaches address classifications of actions,
including image structure-based methods,
posebased systems, model-based approaches, and
example-based methods. The pose-based
method trains each pose using an annotated 3D
image [
[
Moreover, a single person activity can also
be recognized by using smartphone sensors and
wearable sensors; the smartphone-based
approach uses sensors that are inbuilt in the
device, such as accelerometer and gyroscope, to
identify activity, whereas the wearable
sensorbased approach requires the sensors to be
attached on the subject body to collect action
information. [
Our approach to activity recognition and classification consists of two sequential tasks, pose estimation from images, and then the classification of the activities using extracted pose key points as input with the help of classification algorithms such as logistic regression, support vector machine, decision tree, etc. Figure 1 shows the architecture of the proposed approach.
Human Pose Estimation is the task of extracting the body's skeletal key points and joints locations corresponding to the human body parts. It uses all those key points and joints to associate the two-dimensional structure of the human body. In this work, we have used the OpenPose framework for estimating the pose from the input image.
In OpenPose, the image is sent over the CNN output network to get the features from input. The feature map is then processed in the multi-stage CNN sequential layers to generate (PAF) Part Affinity Fields and Confidence Map. The Partial Affinity Fields and Confidence map generated above are passes through a bipartite graph matching algorithm to capture human posture in the image. Figure 2 shows the OpenPose pipeline.
It contains two-dimensional vectors that encode the body part's positions and orientations in an image. It encrypts your data in the form of a double link be-tween body parts.
L = (L1, L2, L3 …. Lc) (1)
Lc∈Rw*h*2 c ∈ {1 … C}, where C is the total number of limbs, R is the real number, L is the set of part affinity field maps, and w x h is the dimension of each map in the set L.
It is a two-dimensional representation of the belief that a particular part of the body can be placed on a specific pixel.
S = (S1, S2, S3 …. Sj) (2)
Sj∈Rw*h j ∈ {1 … J}, where J is the total number of body parts, R is the real number, and S is the set of confidence maps.
The number of keypoints detected through OpenPose dependent upon the dataset has been trained. In this work, the COCO dataset having 18 different body key points (see Figure 3) R_Ankle, R_Knee, R_Wrist, L_Wrist, R_Shoulder, L_Shoulder, L_Ankle, L_Ear, R_Ear, R_Elbow, L_Elbow, L_Knee, L_Eye, R_Eye, R_Hip, L_Hip, Nose, and Neck is used.
We formulate the activity classification problem as a multiclass classification problem, which can be modeled using various machine learning regression and classification algorithms. The classification algorithm takes 18 body keypoints (x-axis and y-axis values of each point) as input for our model's training and testing. We used a supervised learning approach as our dataset contains body keypoints with an activity label. Among all the algorithms, we use multiple logistic regression, and random forest provides significantly greater accuracy.
OpenPose uses the COCO keypoints
detection dataset for the pose estimation task,
which contains more than 200K images labeled
with keypoints [
For performance evaluation, Recall, Precision, and F1-score are used in this experiment. We have also shown the Confusion Matrix of some classifiers. 5.1.
Precision(P) is the ratio of the number of true positives (Tp) to the sum of false positives (Fp) and true positives. It can also be defined as how many images classified into this class belong to this class.
P = Tp/(Tp+Fp) 5.2.
Recall(R) is the ratio of the number of true positives (Tp) to the sum of false negatives (Fn) and true positives. It can also be defined as how many images that belong to this class are classified into this class.
R = Tp/(Tp+Fn) 5.3.
F1-Score is calculated as the harmonic mean of recall and precision. Eqs.5 calculates it.
F1-Score = 2 (P*R)/(P+R) (5) 5.4.
It is a two-dimensional matrix used to measure the overall performance of the machine learning classification algorithm. In the matrix, each row is associated with the predicted activity class, and each column is associated with the actual activity class. The matrix compares the target activity with the activity predicted by the model. This gives a better idea of what types of errors our classifier has made.
The following five activities are considered for pose estimation and activity recognition and classification: sitting, standing, dancing, laying, and running. The experiments are conducted in Scikit Learn (0.23.1) and Python (3.6.6) in Windows 10 Operating System with Intel i5 Processor 3.40 GHz with 8 GB RAM and using five classification algorithms for activity classification. These algorithms are described below with their confusion matrix. The performance results are provided in Table 1, which shows the recall, precision, and f1-score of various classifiers used in the proposed approach.
This algorithm is based on supervised learning, and it is used in classification problems. In this work, multiple logistic regression is used for classifying activities, and 'sag' is used as a solver because it solves only L2 regularization with primal formulation or no regularization and Uses dummy variables to represent the categorical outcome.
K-nearest neighbor (KNN) is a supervised machine learning algorithm used for classification, and it's a non-parametric, lazy algorithm. Despite this simplicity, we got very competitive results that are one reason for using this algorithm in our work. We used different values for k and got the highest accuracy in 5. The distance function(d) used in this algorithm is given in Eqs.6 and for the confusion matrix (see Figure5).
d(p,q) = √ ∑ (qi - pi)2 (6) where p, q are vectors containing keypoints of two different images and i=1 … n.
It also comes under supervised learning algorithms and is mainly used in classification and regression problems. We plotted all available data as points in two-dimensional space. The classification is done by finding a hyperplane that provides similarly different outputs between the two classes. The confusion matrix is provided in Figure6.
The decision tree comes under supervised learning. It is the most powerful and accepted tool for prediction and classification. This algorithm uses learning to predict a target pose's activity and make decisions from previously trained data. Predictions for activities are made from the root of the tree. The record attribute value is compared to the root attribute value. The confusion matrix is given in Figure7.
Random decision forest is a supervised learning algorithm, and it is an ensemble learning method for classification and regression. It is also one of the most used and popular algorithms because it gives better results without tuning hyper-parameter. It creates multiple decision trees and selects the best solution using voting. We use a random forest because it predicts activity with good accuracy and runs efficiently even for big datasets. The confusion matrix is shown in Figure8.
In this study, we proposed our approach for human activity recognition from still images by extracting the skeletal coordinate information (pose) using OpenPose API and then further utilizing this pose information to classify activity with the help of a supervised machine learning algorithm. We prepared our dataset for this work, which contains five different activities, viz. sitting, standing, laying, dancing, running. We have used five algorithms (Logistic Regression, SVM, KNN, Random Forest, and Decision Tree) to find better results for our model. From our experiment results, we observed that Multiple Logistic Regression, SVM, and Random Forest are showing the highest accuracy of 80.72%, 80.43%, and 80.75%, respectively, and the other two algorithms KNN and Decision Tree, are underperforming. We have shown accuracies of some recent researches on HAR in table 2.
Although much research has already been done to a certain extent to deal with the activity recognition problem, more convincing actions must be taken. In practice, there are a lot of different activities that humans use to perform in everyday life. Detecting all of them isn't an easy task because it requires a very large dataset to train the model. Although the dataset is not the only problem, the definition and diversity of activities also make it more complicated for machines to understand. Some more activities can be added to extend the scope and usefulness of the work in the future. Besides adding activities, we can apply some data preprocessing techniques for handling missing keypoints of the body. We can also experiment with some other machine learning algorithms that can provide better results.
algorithm with feedforward CNN
Accuracy (%) 77.0