Human Sport Activities Recognition and Registration from Portable Device Bernardas Zokas, Mantas Lukoševičius Department of Software Engineering Kaunas University of Technology Kaunas, Lithuania zokas.bernardas@ktu.edu, mantas.lukosevicius@ktu.lt Abstract—The drop in costs of hardware prices have led to gyroscope data for recognition. significant changes in the size of various processors and sensors in smartphones. These devices come with a big range of new, Smartphone sensor-based activity recognition topic is not precise measurement taking tools and multi-location sensors new. Our work is slightly different from most previous works (distance sensor, accelerometer, gyroscope, magnetometer, because we use already built commercial mass-marketed camera and lighting sensors). This has opened the door for new device rather than a research-only device. Instead of five or smart device apps that can use data mining applications relying more devices placed on different parts of the human body we on sensor data. One of the main uses is the recognition of human are using only one, at specific location. Our work goal is to movements. In this study, we propose a recognition and tracking recognize specific sport activities and count their repetitions, method in sports activities such as push-ups, sit-ups and squats while performing and storing all recognitions and calculations using only smartphone sensors and a machine learning to a single device. The classifier was built using only one algorithm. The key location for the smartphone is the upper part person’s sport activity data, but after processing and extracting of the user’s left arm. To collect the data and produce features for features we created a universal classifier model, which is classifying sports activities, the motion data from accelerometer independent from the user; the key factor is location and and gyroscope sensors is used. The features are made of two position of the smartphone. sliding windows and additional data processing which renders our classifier even more versatile. Fast response time, light- Our work has several features. Since we tested our classifier weight and accurate sport recognition can be used in mobile model in a real world environment with real people we can start applications like our Home Workout Fitness Tracker which can collecting more data and improve our model for more sports process all the data in real time and create a real time sports activities, as a solution we can see improvement in pull-ups activities tracking system. recognition and offer our app to a larger audience of users. In this way, our classifier would be able to recognize push-ups, Keywords—human activity recognition, machine learning, squats, sit-ups and pull-ups. For such an application, there are sensors, Android application, accelerometer, gyroscope, push-ups, almost no limits in functionality growth like joining heart rate squats, sit-ups. monitor, GPS workouts and voice couching program, etc. Secondly, we can offer the classifier model and dataset that we I. INTRODUCTION developed for further future researches as a solid foundation to Currently, more and more people are filling their lives with start with. sports activities. For this purpose, smartphones are used to In the following paragraphs, we discuss the related work assist in capturing achievements of sport activities. Lower costs (paragraph 2), describe our dataset collection, and describe our in making hardware have led to significant changes in the size, classifier model and an approach which we use to recognize capabilities and functionality of various processors and sensors activity from sensor data (paragraph 3). As a result of the in smartphones. These devices come with a wide range of new, research, we will show our experiment data and the capabilities precise measurement taking tools and multi-location sensors of our application (paragraph 4-5). like cellular radio, Wi-Fi radio, Bluetooth radio, microphone, cameras, GPS, accelerometer, gyroscope, compass, light and proximity sensors [2, 13]. This has opened the door to new II. RELATED WORK smartphone apps that use different sensor data. One of the main Human activity recognition is a wide field for science, uses is the recognition of human movements. The growing researches and data mining. Human activity recognition can be presence of sport activity recognition and data capturing divided in two main groups of video sensor based activity gadgets still has a low market share. A common problem for recognition (VSAR) and physical sensor based activity people who are interested in their health and staying in shape is recognition (PSAR) [1]. According to the review (PSAR) can how to log and track sport activities during the workout. be split in to two smaller groups of wearable sensors and object Therefore we came up with an idea of a smartphone application usage based activity recognition (WSAR and OUAR). Our – a sport activity tracker, which uses tri-axial accelerometer and research will uses wearable sensors embedded on the smartphone itself. In the other article of Activity Recognition Copyright held by the author(s). 61 on Mobile Phones [2] the author made a review about available we can see User motion (human arm movement during sport sensors on the mobile phones. This review also provides main activity) which is captured using smartphone sensors scope of activity recognition, methods of solving activity (accelerometer, gyroscope and magnetometer) and logged into recognition problems, extracting features and basic algorithms smartphone’s memory, then collected sensor data is send to for applying data mining. We can specify that our research in Processing where main features are extracted. The next step the category of using wearable sensors (in our case sensors on leads to Machine Learning Algorithm where extracted features the smartphone) for human activity recognition. are classified and prediction is made for estimating sport activity. Having the same prediction which continuous for Activity recognition using smartphones has a growing certain amount of time the Activity Repetition Counter counts potential in the research of data mining since smartphones and triggers about new counted sport activity repetition and all become equipped with a wide variety of sensors. The most the results are send to GUI where user is able to see his effective sensors for activity recognition and data mining is progress. The main steps are explained in details below: Sensor accelerometer and little less used in the field are gyroscope and – Section C; Preprocessing – Section D; Machine Learning magnetometer. Each of those sensors producing three Algorithm – Section E; Activity Repetition Counter – Section dimensional motion data. There are many studies done using F. these smartphone sensors for specific uses or applications, our project is not an exception. One of them is very similar [5], where two students were solving almost the same problem. They were making push-ups and non-push-ups recognition classifier using Support Vector Machine and Multilayer Perceptron. For feature extraction they used sliding window Fig. 1. Structure of the sports activity recognition system method with 1s, 2s, 4s and 8s length windows. They were solving not only push-up recognition problem but also introduced classifier of squats and sit-ups as a non-push-ups B. System Environment class. They achieved 98% classification rate for push-ups. We have chosen Android-based smartphones as the Other researchers [3-4, 6, 8] were solving walking, slow platform for our project. The Android operating system is free, walking, jogging, going up stairs, going down stairs, sitting and open-source, easy to program, and dominant in the standing recognition problems. In all these papers authors were smartphones market. For the project we will use 5.0 and higher using Weka (Waikato Environment for Knowledge Analysis) versions of Android operating system, which gives us more toolkit [9] and analyzed existing machine learning algorithms. features to work with and much bigger computing power. Mostly in all researches the best recognition rate of over 90% Furthermore, smartphones with higher OS versions have more was produced by the Multilayer Perceptron algorithm. Data precise sensors. Because of big computing power and built-in were collected using smartphone accelerometer and gyroscope sensors, smartphones are an ideal platform for our application; tri-axial sensors. Features for training set were produced using we also require no internet access or additional machine for sliding window methods together examining and extracting calculations. The collected data can be stored directly in the values of mean, standard deviation, mean absolute deviation, smartphone’s memory, because the Android OS provides a time between peaks and the resultant magnitude. built-in SQLite [12] database and big storage capacity. Similar work but using different machine learning C. Data Collecting algorithm was done in a study [7] where authors were solving the problem of walking, jogging, running, going upstairs and To begin with, first we need to decide which part of the going downstairs activity recognition. Recognition rate of 94% human body is the best for locating the smart phone. We are was achieved using DTW (Dynamic Time Warping) [10] collecting sport activities data such as push-ups, sit-ups, and algorithm simply by matching templates of the accelerometer squats workout, we need to locate smartphone [14] on human sensor signal data. The strength of this method is, that it does body so we can measure all of the listed activities. After some not depend on the length of the signal. In this case specific experimenting we found out that the most sensitive place for accelerometer signal can be shorter or longer than its template. taking measurements with smartphone on human body is left In this work as an additional feature was introduced context upper arm next to the shoulder (left arm was chosen because of filtering, which was done by measuring user’s heart rate and better smartphone GUI control with the right hand) with the barometer readings when he was performing the activity. screen facing outwards. We mounted the smartphone using duct tape instead of an armband for a better grip. For data collecting smartphone accelerometer, gyroscope and III. METHODOLOGY magnetometer sensor’s data is captured with the Android In this section we describe our main task of sport activity application designed to access to Android sensor event services recognition and all the processes from start to finish of where we read sensor data. Because we are reading three tri- performing this task. In the section Overview we explain the axial sensors, in total we have nine different arm motion signals algorithm of our sport activity recognition application. (magnetometer x, y, z axis, gyroscope x, y, z axis and magnetometer x, y, z axis). Data reading frequency is 10Hz: A. Overview every 100ms a new reading is made. Since we have limited resources of processing power in smartphone, we decided to To begin with at first we would like to introduce our use lower frequency of sensor data readings. Each sensor algorithm structure which is shown in the Fig. 1. In this figure 62 reading is added to the array list with an included timestamp. order to make accurate classifier we need to label only those At the end of sport activity data logging we stop our application signal parts which contain repetition and other must be labeled and save all collected data into a .csv format file located in as false data. To do so we created simple program with smartphone’s external data storage with a specified file name of MATLAB [15] for filtering and labeling signal data. The sport activity, to easily access it with the computer. algorithm is very simple, using specific parameters of signal height, median, and signal length, we label those signal parts D. Features Extraction where it meets all parameter values. An example of the signal In previous step we collected our sensor data and now we labeling is shown in the Fig. 5. Green vertical line is start of the need to extract some features to be able to make a training filtered signal, red vertical line is middle of the filtered signal dataset and to train a classifier. During examination of our and blue vertical line is end of the filtered signal. During lots collected data, we noticed that some of the sensor’s signals are and lots of experimentations by filtering repetitions and making unreliable. This is happening because sport activity repetition classifiers we came up with the solution of using two fixed have a specific pattern, which can be seen in some of the signal lengths one of 13 samples and one of 25 samples for sensors signals, but this pattern is not visible in all of the repetition filtration. That way we also need two classifiers sensors signals. Some other signals have no pattern at all, for instead of one, because two classifiers are more versatile for example all of the magnetometer signals were not matching the short and long signal sample ranges. We separate all collected pattern and were unusable. Some of the signals were duplicated data files into two groups of short and long repetitions and like accelerometer y and z axis. And some of the signals were applied labeling algorithm for each of them. reliable for one sport activity but not for another. We needed to select only those sensor (magnetometer x, y, z axis, gyroscope x, y, z axis and magnetometer x, y, z axis) signals which were useful for all sport activities, matched pattern and were most promising, also we need not to overload the smartphone processor. After time consuming and frustrating experimentation with the different combinations of signals we found out that the most appropriate signals are accelerometer x, z axis and gyroscope z axis. The problem was that we have different sport activities and at some point not all sensor’s axis were used so the signal readings were almost strait line. Some Fig. 5. Labeled signal example after filtration of the measurements taken in the process are shown in the Fig. 2-4, where we can see different sport activities and three The next question that we faced is how to train the classifier chosen signals (accelerometer x and z axis and gyroscope z with the signal data. We cannot use the whole signal and also axis) for later data processing. we cannot use only one signal sample for training the classifier. The idea is to split signal into small parts of 10 samples for 1 second signal classifier and 20 samples for 2 second signal classifier which came from the dataset analysis where we spotted that most of the sport activities’ repetitions are in the interval of 0.5 to 3 seconds. The best way to split signal data is by using a Sliding Window (SW) method. Our SW consists of 10 samples in length, for each iteration we will select 10 rows Fig. 2. Example of push-ups signals. at a time and transform it to 30 columns (30 columns because we have three different signals for each row) and after each iteration we will move one row lower. The same step we will repeat for 20 sample length SW, but instead 10 rows we will select 20. E. Machine Learning Algorithms Comparison Fig. 3. Example of squats signals Three sport activities were studied as listed above. We extracted features and labeled them to the classes of separate different sport activities, in total we have two training datasets for our machine learning classification problem. One dataset for short sport activities repetitions in average of 1 second time duration and one for longer sport activities repetitions in average of 2 second time duration. We performed and Fig. 4. Example of sit-ups signals evaluated the performance of the following classifiers available in the Weka (Waikato Environment for Knowledge Analysis) Since we have unlabeled data of accelerometer x, z axis and toolkit [9], which are: Multilayer Perceptron, Random Forest, gyroscope z axis, we need to label class for each activity. We Simple Logistic and Logit Boost. Classifiers were trained and have three activities and lots of false data in between tested using a 10-fold cross validation method on the training repetitions. From previous experimentations we learned, that in datasets using default options. The summary of the trained and 63 tested classifier with both datasets recognition rate for sport The additional repetition counter filtration is called as a activities is shown in the TABLE I. The most promising are safety feature. Furthermore, we have two classifiers instead of Multilayer Perceptron and Random Forest. Random Forest one universal and it provides us more chances to correctly have advantage in training time and recognition rate for both 1 recognize sport activities. and 2 second classifier. In the other hand Multilayer Perceptron is losing in recognition rate for both classifiers and also in training time which is significant longer. Other two classifier methods Simple Logistic and Logit Boost are slightly lower in recognition rate, but still take less time to train than Multilayer Perceptron. TABLE I. TABLE OF MACHINE LEARNING CLASSIFIERS TEST RESULTS Recognition rate, % Machine Learning Classifier 1 second 2 seconds Multilayer Perceptron 97.144 96.7 Simple Logistic 97.06 95.837 Logit Boost 96.56 94.885 Random Forest 97.582 97.252 Fig. 6. Random Forest and Multilayer Perceptron prediction comparison F. Sport Activity Repetition Counter Looking at the result the best candidate would be Random Forest, but we decided to use Multilayer Perceptron. It seems to IV. RESULTS be, that even if perceptron is not so accurate in overall The summary of results for our activity recognition recognition rate, but in real world test it shown exact prediction experiments are presented in the TABLE II. This table specifies for the current class comparing with Random Forest. Test was the experiment made in real world environment with real performed using 2 second signal classifier to classify 1 second humans. Experiment consists of 9 subjects who were asked to test dataset. A Multilayer Perceptron and Random Forest perform specific workouts (push-ups, squats and sit-ups) and recognition accuracy test on labeled dataset is shown in the Fig. for each of the workout subject needed to do 10 repetitions of 6. We can see that Multilayer Perceptron gives us almost the fast (from 0.5s to 1s), normal (from 1s to 2s) and slow (from 2s same prediction like one labeled in the training dataset. to 3s) repetition speeds. Before performing sport activities However Random Forest gives us only spikes that are far away recognition test we provide short tutorial to the participants of from the required prediction. We must keep in mind that our how to perform each workout correctly. Then we mounted goal is not just to recognize sport activities, but it is also to smartphone with the installed application (the application calculate repetitions and the closer we are to the labeled dataset screen shots of main menu, history window and main sport pattern the better it is. Having spikes or one sample predictions activity tracker window are represented in the Fig. 7) on the per repetition like Random Forest have, we can’t subject’s left upper arm. To calibrate smartphone’s exact programmatically correct it or filter out some of the features, location on the arm we asked to stretch out participant’s hands but with Multilayer Perceptron it is possible. For example, to in horizontal position with the palms facing down. All the count one repetition we must have at least 3 continuous participants were men of different heights and weights. predictions in a row without noise. Even if classifier is failing at same point, we still can rely on the average of continuous activity’s predictions and improve repetition counter. TABLE II. TABLE OF SPORT ACTIVITY RECOGNITION TEST WITH REAL SUBJECTS Push-ups Squats Sit-ups Participants Fast Normal Slow Fast Normal Slow Fast Normal Slow 1. 10 10 10 10 10 10 9 10 10 2. 8 10 10 8 10 10 7 9 10 3. 9 10 10 8 10 10 7 10 10 4. 10 10 9 10 10 10 8 10 10 5. 10 10 10 10 10 10 9 10 10 6. 10 9 8 10 9 9 10 10 10 7. 10 10 10 10 10 9 10 10 9 8. 10 10 9 10 10 8 10 10 9 9. 8 10 10 9 10 10 8 10 10 Recognition rate, % 94.444 98.889 95.556 94.444 98.889 96.667 86.667 98.889 97.778 64 existing sport activity recognition algorithm has perfect core for adding more features by its flexibility and light framework. During the testing smartphone showed no visual lagging or overstress in performing the recognitions, it can be also used in a background mode when user can use other applications at the same time. REFERENCES [1] D. Guan, T. Ma, W. Yuan, Y. Lee and A. M. J. Sarkar, “Review of Sensor-based Activity Recognition Systems”, IETE Technical Review, 28:5, 418-433, 2011. [2] Incel O.D., Kose M., Ersoy C. “A Review and Taxonomy of Activity Recognition on Mobile Phones”, BioNanoScience, 3:145-171, 2013. Fig. 7. Sport activity recognition application print screens [3] J. R. Kwapiz, G.M. Weiss and S. A. Moore, “Activity Recognition Using Cell Phone Accelerometers”, ACM SIGKDD Explorations Newsletter, 12:2, 74-82, 2010. V. CONCLUSION [4] A. Bayat, M. Pomplun, D. A. Tran, “A Study on Human Activity In this study, human sport activity recognition accuracy of Recognition Using Accelerometer Data from Smartphones”, Procedia up to 95% on sport activities such as push-ups, squats and sit- Computer Science 34, 450-457, 2014. ups using a tri-axial accelerometer and gyroscope was obtained. [5] M. Nilsson and H. Wilén, “Push-up Tracking through Smartphone The sensors’ data was collected from one subject by Sensors”, School of Engineering Sciences, 23p, 2016. performing listed sport activities workouts using a smartphone [6] R. Piyare and S. R. Lee, “Dynamic Activity Recognition using as a sensor which was located on the subject’s upper left arm. Smartphone Sensor Data”, 1st International Electronic Conference on Sensors and Applications, 2014. Collected data was separated into two groups of short and long [7] S. M. S. Hasan, M. Masnad, M. M. Khan, H. Mahmud and M. K. Hasan, activity’s repetition performing speeds, labeled, and features “Human Activity Recognition using Smartphone Sensors with Context were extracted. The training datasets were made and two Filtering”, The Ninth International Conference on Advances in classifiers were trained for each of collected data group. Computer-Human Interactions, 2016. Combining two classifiers, smartphone’s sensors and additional [8] M. Shoaib, S. Bosch, O. D. Incel, H. Scholten and P. JM Havinga, filtering for repetition counting the Android application was “Complex Human Activity Recognition Using Smartphone and Wrist- created. To measure sport activity recognition accuracy the test Worn Motion Sensors”, Multidisciplinary Digital Publishing Institute, 16:4, 2016. of nine male subjects was performed and achieved the average [9] Weka machine learning toolkit. of 95.8% sport activity recognition rate. The main benefits of http://www.cs.waikato.ac.nz/ml/index.html. Accessed March 2018. the project are that sport activity recognition is done in [10] Dynamic Time Warping. smartphone by itself, using its own computing power and no web.science.mq.edu.au/~cassidy/comp449/html/ch11s02.html. Accessed additional devices, or servers, or internet access are require. All March 2018. the collected data is processed in real time and have only 2 [11] Damaševičius, R., Napoli, C., Sidekerskienė, T. and Woźniak, M. “IMF seconds time delay at displaying data in GUI, which is limited mode demixing in EMD for jitter analysis.” Journal of Computational of 2 second classifier (longest time for getting 20 samples of Science, vol. 22, pp.240-252, 2017. data). The extracted features are little different from the [12] SQLite. https://www.sqlite.org/about.html. Accessed March 2018. previous works in other papers. The difference occurs that our [13] N. D. Lane, E. Miluzzo, H. Lu, D. Peebles, T. Choudhury, and A. T. method uses raw sensor’s signal data instead of calculating Campbell, “A survey of mobile phone sensing,” IEEE Communications Magazine, vol. 48, no. 9, pp. 140–150, 2010. additional features. During the recognition testing we have noticed that bad or misshape workout repetitions weren’t [14] W. Xu, M. Zhang, A. A. Sawchuk, and M. Sarrafzadeh, “Co-recognition of human activity and sensor location via compressed sensing in counted, which leads us to make conclusion that application wearable body sensor networks,” in Proc. 9th IEEE Int. Conf. Wearable also promotes subject to perform right movements for the sport Implantable Body Sensor Netw., 2012, pp. 124–129. activity. Existing project can be easily improved with [15] MATLAB. https://se.mathworks.com/products/matlab.html. Accessed additional testing with more subjects especially women and March 2018. new classifier can be made with the bigger training dataset [16] Beritelli, F., Capizzi, G., Sciuto, G. L., Napoli, C., & Scaglione, F. made of more than one subject’s data. Furthermore, according (2018). Automatic heart activity diagnosis based on gram polynomials to the plan current application needs to be improved to be and probabilistic neural networks. Biomedical Engineering Letters, vol. 8, issue 1, pp. 77-85, 2018. capable of recognizing pull-ups and etc., which was mentioned [17] Połap, D., Wozniak, M., Napoli, C. and Tramontana, E., 2015. “Is in the application specification. Then it would have more sports swarm intelligence able to create mazes?” International Journal of activities on the list and make application even more advance. Electronics and Telecommunications, 61(4), pp.305-310, 2015. Other feature is to set up additional module for voice coaching [18] Woźniak, M., Połap, D., Napoli, C. and Tramontana, E., Graphic object to make sport activity recognition application more user feature extraction system based on cuckoo search algorithm. Expert friendly by using phrases to describe how much repetitions left, Systems with Applications, 66, pp.20-31, 2016. when the goal is achieved and push user to exercise harder. The [19] 65