-The drop in costs of hardware prices have led to significant changes in the size of various processors and sensors in smartphones. These devices come with a big range of new, precise measurement taking tools and multi-location sensors (distance sensor, accelerometer, gyroscope, magnetometer, camera and lighting sensors). This has opened the door for new smart device apps that can use data mining applications relying on sensor data. One of the main uses is the recognition of human movements. In this study, we propose a recognition and tracking method in sports activities such as push-ups, sit-ups and squats using only smartphone sensors and a machine learning algorithm. The key location for the smartphone is the upper part of the user's left arm. To collect the data and produce features for classifying sports activities, the motion data from accelerometer and gyroscope sensors is used. The features are made of two sliding windows and additional data processing which renders our classifier even more versatile. Fast response time, lightweight and accurate sport recognition can be used in mobile applications like our Home Workout Fitness Tracker which can process all the data in real time and create a real time sports activities tracking system.
Currently, more and more people are filling their lives with
sports activities. For this purpose, smartphones are used to
assist in capturing achievements of sport activities. Lower costs
in making hardware have led to significant changes in the size,
capabilities and functionality of various processors and sensors
in smartphones. These devices come with a wide range of new,
precise measurement taking tools and multi-location sensors
like cellular radio, Wi-Fi radio, Bluetooth radio, microphone,
cameras, GPS, accelerometer, gyroscope, compass, light and
proximity sensors [
gyroscope data for recognition.
Smartphone sensor-based activity recognition topic is not new. Our work is slightly different from most previous works because we use already built commercial mass-marketed device rather than a research-only device. Instead of five or more devices placed on different parts of the human body we are using only one, at specific location. Our work goal is to recognize specific sport activities and count their repetitions, while performing and storing all recognitions and calculations to a single device. The classifier was built using only one person’s sport activity data, but after processing and extracting features we created a universal classifier model, which is independent from the user; the key factor is location and position of the smartphone.
Our work has several features. Since we tested our classifier model in a real world environment with real people we can start collecting more data and improve our model for more sports activities, as a solution we can see improvement in pull-ups recognition and offer our app to a larger audience of users. In this way, our classifier would be able to recognize push-ups, squats, sit-ups and pull-ups. For such an application, there are almost no limits in functionality growth like joining heart rate monitor, GPS workouts and voice couching program, etc. Secondly, we can offer the classifier model and dataset that we developed for further future researches as a solid foundation to start with.
In the following paragraphs, we discuss the related work (paragraph 2), describe our dataset collection, and describe our classifier model and an approach which we use to recognize activity from sensor data (paragraph 3). As a result of the research, we will show our experiment data and the capabilities of our application (paragraph 4-5).
Human activity recognition is a wide field for science,
researches and data mining. Human activity recognition can be
divided in two main groups of video sensor based activity
recognition (VSAR) and physical sensor based activity
recognition (PSAR) [
Activity recognition using smartphones has a growing
potential in the research of data mining since smartphones
become equipped with a wide variety of sensors. The most
effective sensors for activity recognition and data mining is
accelerometer and little less used in the field are gyroscope and
magnetometer. Each of those sensors producing three
dimensional motion data. There are many studies done using
these smartphone sensors for specific uses or applications, our
project is not an exception. One of them is very similar [
Other researchers [
Similar work but using different machine learning
algorithm was done in a study [
In this section we describe our main task of sport activity recognition and all the processes from start to finish of performing this task. In the section Overview we explain the algorithm of our sport activity recognition application.
To begin with at first we would like to introduce our algorithm structure which is shown in the Fig. 1. In this figure we can see User motion (human arm movement during sport activity) which is captured using smartphone sensors (accelerometer, gyroscope and magnetometer) and logged into smartphone’s memory, then collected sensor data is send to Processing where main features are extracted. The next step leads to Machine Learning Algorithm where extracted features are classified and prediction is made for estimating sport activity. Having the same prediction which continuous for certain amount of time the Activity Repetition Counter counts and triggers about new counted sport activity repetition and all the results are send to GUI where user is able to see his progress. The main steps are explained in details below: Sensor – Section C; Preprocessing – Section D; Machine Learning Algorithm – Section E; Activity Repetition Counter – Section F.
We have chosen Android-based smartphones as the
platform for our project. The Android operating system is free,
open-source, easy to program, and dominant in the
smartphones market. For the project we will use 5.0 and higher
versions of Android operating system, which gives us more
features to work with and much bigger computing power.
Furthermore, smartphones with higher OS versions have more
precise sensors. Because of big computing power and built-in
sensors, smartphones are an ideal platform for our application;
we also require no internet access or additional machine for
calculations. The collected data can be stored directly in the
smartphone’s memory, because the Android OS provides a
built-in SQLite [
To begin with, first we need to decide which part of the
human body is the best for locating the smart phone. We are
collecting sport activities data such as push-ups, sit-ups, and
squats workout, we need to locate smartphone [
In previous step we collected our sensor data and now we need to extract some features to be able to make a training dataset and to train a classifier. During examination of our collected data, we noticed that some of the sensor’s signals are unreliable. This is happening because sport activity repetition have a specific pattern, which can be seen in some of the sensors signals, but this pattern is not visible in all of the sensors signals. Some other signals have no pattern at all, for example all of the magnetometer signals were not matching the pattern and were unusable. Some of the signals were duplicated like accelerometer y and z axis. And some of the signals were reliable for one sport activity but not for another. We needed to select only those sensor (magnetometer x, y, z axis, gyroscope x, y, z axis and magnetometer x, y, z axis) signals which were useful for all sport activities, matched pattern and were most promising, also we need not to overload the smartphone processor. After time consuming and frustrating experimentation with the different combinations of signals we found out that the most appropriate signals are accelerometer x, z axis and gyroscope z axis. The problem was that we have different sport activities and at some point not all sensor’s axis were used so the signal readings were almost strait line. Some of the measurements taken in the process are shown in the Fig. 2-4, where we can see different sport activities and three chosen signals (accelerometer x and z axis and gyroscope z axis) for later data processing.
Fig. 2. Example of push-ups signals.
Since we have unlabeled data of accelerometer x, z axis and
gyroscope z axis, we need to label class for each activity. We
have three activities and lots of false data in between
repetitions. From previous experimentations we learned, that in
order to make accurate classifier we need to label only those
signal parts which contain repetition and other must be labeled
as false data. To do so we created simple program with
MATLAB [
The next question that we faced is how to train the classifier with the signal data. We cannot use the whole signal and also we cannot use only one signal sample for training the classifier. The idea is to split signal into small parts of 10 samples for 1 second signal classifier and 20 samples for 2 second signal classifier which came from the dataset analysis where we spotted that most of the sport activities’ repetitions are in the interval of 0.5 to 3 seconds. The best way to split signal data is by using a Sliding Window (SW) method. Our SW consists of 10 samples in length, for each iteration we will select 10 rows at a time and transform it to 30 columns (30 columns because we have three different signals for each row) and after each iteration we will move one row lower. The same step we will repeat for 20 sample length SW, but instead 10 rows we will select 20.
Three sport activities were studied as listed above. We
extracted features and labeled them to the classes of separate
different sport activities, in total we have two training datasets
for our machine learning classification problem. One dataset
for short sport activities repetitions in average of 1 second time
duration and one for longer sport activities repetitions in
average of 2 second time duration. We performed and
evaluated the performance of the following classifiers available
in the Weka (Waikato Environment for Knowledge Analysis)
toolkit [
TABLE OF MACHINE LEARNING CLASSIFIERS TEST RESULTS Machine Learning Classifier
Multilayer Perceptron
Simple Logistic
Logit Boost Random Forest
Looking at the result the best candidate would be Random Forest, but we decided to use Multilayer Perceptron. It seems to be, that even if perceptron is not so accurate in overall recognition rate, but in real world test it shown exact prediction for the current class comparing with Random Forest. Test was performed using 2 second signal classifier to classify 1 second test dataset. A Multilayer Perceptron and Random Forest recognition accuracy test on labeled dataset is shown in the Fig. 6. We can see that Multilayer Perceptron gives us almost the same prediction like one labeled in the training dataset. However Random Forest gives us only spikes that are far away from the required prediction. We must keep in mind that our goal is not just to recognize sport activities, but it is also to calculate repetitions and the closer we are to the labeled dataset pattern the better it is. Having spikes or one sample predictions per repetition like Random Forest have, we can’t programmatically correct it or filter out some of the features, but with Multilayer Perceptron it is possible. For example, to count one repetition we must have at least 3 continuous predictions in a row without noise. Even if classifier is failing at same point, we still can rely on the average of continuous activity’s predictions and improve repetition counter.
Participants 1. 2. 3. 4. 5. 6. 7. 8.
9.
Recognition rate, %
Fast 10 8 9 10 10 10 10 10 8 94.444
Push-ups Normal 10 10 10 10 10 9 10 10 10 98.889
The additional repetition counter filtration is called as a safety feature. Furthermore, we have two classifiers instead of one universal and it provides us more chances to correctly recognize sport activities.
The summary of results for our activity recognition experiments are presented in the TABLE II. This table specifies the experiment made in real world environment with real humans. Experiment consists of 9 subjects who were asked to perform specific workouts (push-ups, squats and sit-ups) and for each of the workout subject needed to do 10 repetitions of fast (from 0.5s to 1s), normal (from 1s to 2s) and slow (from 2s to 3s) repetition speeds. Before performing sport activities recognition test we provide short tutorial to the participants of how to perform each workout correctly. Then we mounted smartphone with the installed application (the application screen shots of main menu, history window and main sport activity tracker window are represented in the Fig. 7) on the subject’s left upper arm. To calibrate smartphone’s exact location on the arm we asked to stretch out participant’s hands in horizontal position with the palms facing down. All the participants were men of different heights and weights.
TABLE OF SPORT ACTIVITY RECOGNITION TEST WITH REAL SUBJECTS
In this study, human sport activity recognition accuracy of up to 95% on sport activities such as push-ups, squats and situps using a tri-axial accelerometer and gyroscope was obtained. The sensors’ data was collected from one subject by performing listed sport activities workouts using a smartphone as a sensor which was located on the subject’s upper left arm. Collected data was separated into two groups of short and long activity’s repetition performing speeds, labeled, and features were extracted. The training datasets were made and two classifiers were trained for each of collected data group. Combining two classifiers, smartphone’s sensors and additional filtering for repetition counting the Android application was created. To measure sport activity recognition accuracy the test of nine male subjects was performed and achieved the average of 95.8% sport activity recognition rate. The main benefits of the project are that sport activity recognition is done in smartphone by itself, using its own computing power and no additional devices, or servers, or internet access are require. All the collected data is processed in real time and have only 2 seconds time delay at displaying data in GUI, which is limited of 2 second classifier (longest time for getting 20 samples of data). The extracted features are little different from the previous works in other papers. The difference occurs that our method uses raw sensor’s signal data instead of calculating additional features. During the recognition testing we have noticed that bad or misshape workout repetitions weren’t counted, which leads us to make conclusion that application also promotes subject to perform right movements for the sport activity. Existing project can be easily improved with additional testing with more subjects especially women and new classifier can be made with the bigger training dataset made of more than one subject’s data. Furthermore, according to the plan current application needs to be improved to be capable of recognizing pull-ups and etc., which was mentioned in the application specification. Then it would have more sports activities on the list and make application even more advance. Other feature is to set up additional module for voice coaching to make sport activity recognition application more user friendly by using phrases to describe how much repetitions left, when the goal is achieved and push user to exercise harder. The [19] existing sport activity recognition algorithm has perfect core for adding more features by its flexibility and light framework. During the testing smartphone showed no visual lagging or overstress in performing the recognitions, it can be also used in a background mode when user can use other applications at the same time.