-Human activity recognition is a significant component of many innovative and human-behavior based systems. The ability to recognize various human activities enables the developing of intelligent control system. Usually the task of human activity recognition is mapped to the classification task of images representing person's actions. This paper addresses the problem of human activities' classification using various machine learning methods such as Convolutional Neural Networks, Bag of Features model, Support Vector Machine and K-Nearest Neighbors. This paper provides the comparison study on these methods applied for human activity recognition task using the set of images representing five different categories of daily life activities. The usage of wearable sensors that could improve classification results of human activity recognition is beyond the scope of this research.
Keywords–activity recognition; machine learning; CNN; BoF; KNN; SVM
Recently the human activity recognition problem has
become a significant matter of research. In most of the cases it
has a very explicit practical applicability: human activity
recognition is an integrate part of human behavior-based
system. Nowadays, smart home technologies are getting a lot of
attention because of better care of the residents which is
extremely important for elderly, children or disabled people [
surveillance systems can be indicated as the typical examples
of such kind of systems [
static postures and dynamic motions is not sufficient. One of
the key features of smart system technologies’ task for human
activity recognition is enabling to identify the current activity
considering to the wide range of provided indoor activities.
Fully-autonomous and barely noticeable assisting systems are
becoming more appropriate for daily use than equipment based
on wearable sensors or appliances [
Although these products have been partially successful, their deployment is not practical, limiting the mobility area of the human (e.g., public areas are excluded). Furthermore the wearable motion sensors make human’s movement cumbersome. Additionally, the installation and maintenance of the sensors usually cause high costs. According to these facts, the more practical solutions rely on the combination of video monitoring devices and image classification methods.
Various machine learning technologies are applied for image recognition tasks. Therefore, the major challenge in human activity recognition is to evaluate the reliability of selected technologies. Considering this fact, it is necessary to compare the experimental results obtained using different machine learning approaches. In this paper, four different methods have been chosen for experiments: Convolutional Neural Networks (CNNs), Bag of Features (BoF), Support Vector Machine (SVM) and K-Nearest Neighbors (KNN). Using the same set of images representing human daily life activities these methods have been applied for the image classification into five categories.
The general schema of human activity classification using all four methods mentioned above is presented in Fig. 1.
Set of images
Image selection Category II
... CNN
BoF SVM k-NN Machine learning method application
Depending on the machine learning methods, the different requirements are imposed on images. For example, using CNN, all images must be of the same size, which is usually pretty small (e.g., 224×224×3). KNN classifier may be enhanced by converting images from RGB color model to LAB model, which enables to quantify visual differences of colors and may lead to better results. SVM algorithm is used for image classification if RGB images are converted to grayscale images and then to binary images.
CNN is a deep learning model that obtains complicated hierarchical features via convolutional operation alternating with sub-sampling operation on the raw input images.
Convolutional neural networks have become one of the most
widely spread models of deep learning and have shown a very
Input layer
high accuracy results in various image recognition tasks [
Convolutions
Max pooling Max pooling
Fully connected ...
Convolutions convolutions, pooling, and fully connected layers (Fig. 2). Convolutional layer computes the output of neurons that are connected to local regions in the input. Pooling layer reduces the spatial size of the representation to reduce the amount of parameters and computation in the network. All these layers are followed by fully connected layers leading into Softmax, which is a final classifier.
The images of the same size a×a×b (where a is the height and width of the image, b is the number of channels) are passed as the input to a convolutional layer. When RGB image is used, b is equal to 3. The convolutional layer has m kernels (or filters) of size c×c×d, where c is smaller than a.
The neurons of the convolutional layer are connected to the sub-regions of the input image (for the first convolutional layer) or the output of the previous layer. Feature map is formed when a filter moves along the input and uses the same set of weights and bias for the convolution. If l is a convolutional layer, the ith feature map Yi(l) is defined using formula: 1( −1) feature map in layer (l-1) with ith feature map in layer l and m1(l1) is the amount of feature maps in layer l-1.
The convolutional layer is followed by an activation function. Rectified linear unit is represented by ReLU layer. ReLU is a function defined as: ( ) = max(0,
( −1)) ( ) = ( −1), ℎ ( −1) ≥ 0
( ) = 0, ℎ ( −1) < 0 (1) (2) (3) (4)
Output layer ) ( , ) )2) ′ = ( +
(,) = ∗
ℎ ( + ∑ ( )
,
min( −1, +
=max(0, −2)
)
2 (
Cross
channel
normalization
(local
response
normalization) layer follows ReLU layer. This layer replaces all
elements with normalized values. The normalized value ‘for
each element x is defined as:
where K, α and β are hyper-parameters in the normalization, s
is the sum of squares of the elements in the normalization
window [
Pooling layers follow convolutional layers and summarize the outputs of near groups of neurons in the same kernel map. The neighborhoods summarized by adjacent pooling units do not overlap. Max-pooling layer returns the maximum values of the input‘s rectangular regions and respectively, averagepooling layer returns average values.
The convolutional layer is followed by a particular amount of fully connected layers. The aim of the convolutional layer is to determine large patterns using the combinations of the features known from previous layers. In order to classify the images, the last fully connected layer combines the identified patterns. The final fully connected layer is followed by Softmax layer and classification (output) layer. In the classification layer, the network takes the values from the Softmax function and assigns each input to one of classes. (5) (6)
Three of CNN architectures have been selected for
experiments in this paper: AlexNet [
B.
Bag of Features encodes the image features into a representation suitable for image classification. This technique is also often referred to as Bag of Words, because it uses image features as visual words represented as image. The features (which sometimes can be general, such as color, texture or shape) are used to find the similarities between images (Fig. 3). histogram .... ....
Classification Extracted features
Extracted features
BoF has shown the promising results (over 80% of
accuracy) in the tasks of action recognition in video sequences
[
Support Vector Machine (SVM) belongs to the class of
machine learning algorithms called kernel methods. It is one of
the best known methods in pattern classification and image
classification. The SVM method was designed to be applied
only for two-class problems. In the context of human activity
classification problem, usually there are more than two possible
classes (categories). Depending on this fact, it is extremely
important to use modified SVM, which can be applied for
multiclass classification. Two main approaches have been
suggested to solve this problem [
For recognition of very simple Daily Living activities
(siting, standing, walking) by carrying a waist-mounted
smartphone with embedded inertial sensors, multiclass SVM
(“one against all” approach) has shown an overall accuracy of
more than 90%. However, the accuracy results are much lower
(71.63%) trying to classify more complex activities [
matched features in different sequences of actions (it may
appear that running for some people is similar to the jogging for
the others). However, employing 3D trajectories of body joints
obtained by Kinect can provide remarkably good results of
accuracy 90.57% [
K-Nearest Neighbors approach is a machine learning algorithm, which is often used for classifying objects based on the most similar training samples in the feature space. The classification is based on distance between a set of input data points and training points. Various metrics can be used to determine the distance (Euclidean distance, Mahalanobis distance, Spearman distance and etc.). KNN search enables to find k closest points in A (a set of n points) to a set of query points, when A and distance function are given. This algorithm is widely used in image processing and classification tasks.
The objects are classified according to the features of its k nearest neighbors by majority vote. Training process consists of storing feature vectors and labels of the training images. During the classification, the unlabelled query point is simply assigned to the label of its k nearest neighbors.
The performance of KNN application for classification of
human activities particularly was examined using uni-axial
sensors (sternum, wrist, thigh, and lower leg) [
Human activity classification results for the particular
method are often represented as confusion matrix (n is
equal to the amount of categories). Confusion matrix is such
that the element is the amount of instances from category i
that were actually classified as category j [
The confusion matrix for binary classification contains four elements (TABLE I): True Positives (TP) represent the amount of positive instances that were classified as positive; True Negatives (TN) represent the amount of negative instances that were classified as negative; False Positives (FP) represent the amount of negative instances that were classified as positive; False Negatives (FN) represent the amount of positive instances that were classified as negative.
The accuracy is widely used metric for the generalization of classification results. This metric is defined using formula:
The confusion matrix and the accuracy can also be used for n categories, where n should be more than 1 (see TABLE II – TABLE VII). In this case, the instance could be positive or negative according to the particular category, e.g., positives might be all instances of category II (e.g., sleeping) while negatives would be all instances other than category II (e.g., other than sleeping).
Despite of considerable amount of scientific research,
human activity recognition only from images is still a very
challenging task due to the background clutter, viewpoint,
lighting, appearance and the rest of wide range aspects.
Moreover, the similarities between different human actions
make the classification even more challenging. The same
activity may be expressed by people who have completely
different appearance, body movements, postures and habits
[
In this paper the experimental scenario including five possible categories depending on the type of human activity has been created (Fig. 4). The activities are supposed to be performed in home or office areas. Category I relates to the situation when the people are communicating. Category II is assigned to the situation when the people are sleeping or having a rest. Category III represents empty spaces (human staying temporarily in the selected area). Human’s work at computer, reading, writing or studying is assigned to category IV. Any type of eating or drinking activities are assigned to the category V. Differently from common activities’ images in various recognition tasks, images representing these activities include all other objects naturally appearing while performing the particular activity. Therefore, the accuracy of expected results may not be as high as they are provided in previous researches (especially where additional techniques or methods are included). a) c) b) d)
The image datasets containing 502 images for each category of human activity have been collected. Each dataset has been split into a training set (which contains 400 images for each category) and a test set (which contains 102 images for each category). The data for training and testing has been chosen randomly from the primary datasets.
The experiments of human activities’ classification have
been implemented using MATLAB software and Add-Ons
[
The estimated classification accuracy of human activity recognition task using different image classification methods is presented in TABLE II – TABLE VII. The average accuracy of KNN is the worst one and approaches to 40.98% (although, it is more than twice the probability to choose the correct class randomly). The difference between average accuracy of SVM and Bag of Features is less than 9%. The values are 59.61% and 68.24%, respectively (the probability to choose the correct class using one of these methods is more than 0.5). The use of CNN architectures (AlexNet, CaffeRef and VGG) provides very similar results. Despite this fact, the average accuracy of AlexNet is the best one and approaches to 90.78%.
The experimental results have shown that activities of category III determine the best results of classification for all methods except KNN (Fig. 5).
The recognition of activities belonging to II, IV and V categories are the most complicated, therefore provides the worst results of classification.
methods used to recognize human activities has been performed.
selected in this research, including CNNs, BoF model, SVM and
mentioned methods for human activity recognition only from images using five different categories of daily life activities.
have not been considered. The obtained accuracy results satisfy our expectations, especially taking into account the consideration that images representing these activities include all other objects naturally appearing while performing the particular activity. The average accuracy of image classification using BoF is 68.24%.
59.61%. Based on the experimental results we can conclude that KNN is not an appropriate method for human activity classification, using such complicated pictures of activities and applying classical KNN notation without any improvements or technological supplements. The application of different CNN architectures has revealed very similar high accuracy results, although AlexNet has reached more than 90% average accuracy, which indicates the best score of all applied methods.
analyze the eligibility of different and newly created CNN architectures for the solution of image-based human activity classification problem.