A Method for Activity Recognition Partially Resilient on Mobile Device Orientation Nikola Jajac, Bratislav Predic, Dragan Stojanovic University of Nis, Faculty of Electronic Engineering 18000 Nis, Serbia {nikola.jajac, bratislav.predic, dragan.stojanovic}@elfak.ni.ac.rs ABSTRACT system for recognizing activities could help reduce errors that This paper demonstrates a method for activity recognition arise from previously mentioned methods. Also, such system partially resilient on mobile device orientation, by using data from enables users to go about their daily routines, while the data a mobile phone embedded accelerometer. This method is partially collection and processing are done in the background, and do not resilient on mobile device orientation, in such a way that a mobile interfere with their current activities. device can be rotated around only one axis for an arbitrary angle. In recent years a lot of work has been done on activity recognition The classifier for activity recognition is built using data from one from accelerometer data. Since an accelerometer is a standard part default orientation. This method introduces a calibration phase in of modern mobile devices (like mobile phones and tablets), they which the phone’s orientation is determined. After that, can be used in activity recognition. An advantage in using these accelerometer data is transformed into the default coordinate devices is that they are already commonly used by a lot of people system and further processed. The solution is compared with the that would not have to wear an additional device to perform method that built a classifier using data from multiple activity recognition, which greatly increases the acceptance of orientations. Three classifiers were tested and a high accuracy of such a system. around 90% was achieved for all of them. This paper focuses on recognizing activities from accelerometer data processed at a mobile phone. Activity recognition is 1. INTRODUCTION formulated as a classification problem. This paper considers the The advance of low-cost and low-power sensors has led to their healthcare domain for activity recognition system application. For massive integration into modern mobile devices that have become this reason it is important to recognize physical activities, such as: powerful sensing platforms. By using data from sensors like walking, running, walking up/down stairs etc. Examples of accelerometers, gyroscopes, digital compasses, light sensors etc. it possible users in this domain are the elderly and persons with is possible to describe the context of a mobile device user in much certain disabilities. It can be assumed that such users keep their more detail than by using only location data. A richer description mobile phone relatively (but not completely) fixed. In the paper it of a user context provides better adaptation of mobile content, is assumed that the phone position is fixed, but that the phone services and resources, enabling the user to stay focused on the orientation is only partially fixed. The orientation is partially fixed task at hand. in such a way that the axis that is perpendicular to the phone’s An activity of the user represents an important aspect of a context, screen is parallel to the ground and the phone can be rotated freely because it directly impacts user’s ability to interact with the around that axis. For experiments in this paper, the mobile phone mobile device and applications. By employing the information was worn in the right front pants pocket (as one of the places about the current activity, the mobile device can adapt its where people usually carry mobile phones) and the screen of the interfaces, filter the content it provides, or perform a specified phone was facing the user. The orientation where the bottom side action, to support the user in the best possible way. of the phone is facing the ground is considered the default orientation. First, three classifiers were built using data from The greatest possibilities for application of activity recognition multiple orientations. These classifiers were tested using data systems lay in the healthcare domain. For example, such systems from the same orientations. Then, classifiers were built again, but can be used for elderly care support or for long-term health/fitness using data from the default orientation only. These new classifiers monitoring [1]. Current methods for tracking activities, like were tested using data from multiple orientations, transformed paying a trained observer or relying on self-reporting are time and into the default coordinate system prior to testing, according to resource consuming tasks, and are error prone. An automatic the method proposed in this paper. After that, classifier performances from these two tests were compared. Using results from previously mentioned tests, we created an application for activity recognition that runs on a mobile device in real time and tested the impact that data transformation has on the performance of such an application, specifically in terms of processor load. The processor load is important because if such application is to be accepted by users, it must not significantly decrease battery life, or the performance of the device in everyday tasks. 1 Proceedings IMMoA’13 15 http://www.dbis.rwth-aachen.de/IMMoA2013/ The proposed method represents only an intermediate step in the mobile device, which could decrease acceptability of such a development of a method for activity recognition with no system. restrictions in mobile device orientation. Sun et al. in [5] tried to recognize activities by using acceleration The rest of the paper is structured as follows: section 2 provides data from a mobile phone, in a setting where the position and the an overview of related work on activity recognition. Section 3 orientation of the phone vary. They restrict their hypothesis space describes the process of accelerometer data collection. In sections to 6 possible positions (6 pockets) and 4 orientations of the 4 and 5, mixed orientation data and data reorientation processing mobile phone. The data from all position and orientation approach, for activity recognition resilient on mobile device combinations were collected. The authors added acceleration orientation, are presented and evaluated. Section 6 presents an magnitude at each sample, as an additional sensor reading evaluation of how much the activity recognition application and dimension. By using collected data, several features were data transformation phase participate in the processor load. calculated: mean, variance, correlation, FFT energy and Section 7 gives the conclusions about the paper and outlines plans frequency-domain entropy. Calculated features were used to train for future work. and test SVM (Support Vector Machine) models. Generated SVM model recognizes activities with an accuracy of 93.1% throughout all tested positions and orientations. 2. RELATED WORK In recent years there has been a lot of research related to Thiemjarus in [6] applied a different approach. Accelerometer was recognizing activities from accelerometer data. In [2] authors used mounted on a belt-clip device which was worn by test subjects in data from 5 biaxial accelerometers worn simultaneously on a fixed position on a body, but which could be mounted in 4 different parts of the body. Used accelerometers could detect different orientations. Data was collected by 13 subjects that acceleration up to ±10G. Accelerometers were mounted onto performed a routine comprised of 6 activities. The first step in hoarder boards and firmly attached to different body parts. Data data analysis was device orientation detection. The orientation was collected from 20 subjects performing various everyday tasks detection was also formulated as a classification problem. The without researcher supervision. The following features were features used for orientation detection were mean along Y and Z computed on sliding windows of accelerometer data: mean, axis. Orientation detection was performed for an activity routine energy, frequency-domain entropy and correlation. A number of performance, which contains approximately 5 seconds of data for classifiers were trained and tested with the calculated data, where each tested activity, while activity recognition was window-based. decision trees showed the best result, recognizing activities with The second step was signal transformation using the appropriate an accuracy of 84%. transformation matrix, and the third step was activity recognition itself. Author achieved a subject-independent classification Ravi et al. in [3] attempted to perform activity recognition using a accuracy of 90.9%. single triaxial accelerometer worn near the pelvic region. Data was collected by 2 subjects performing 8 different activities. A possible problem with the approach in [6] is that orientation Similarly to [2] the features were computed using the sliding detection is done on a data set that includes information about all windows technique. Four features were extracted: mean, standard tested activities. The open issue is how this approach can be deviation, energy and correlation. Extracted features were used to applied in a real world scenario, when the user does not perform a train and test 5 base-level classifiers, and in addition to that, 5 specified activity routine. A unifying fact for [5, 6] is that the meta-level classifiers. Authors concluded that meta-level hypothesis space is limited to a number of orientations. Again, the classifiers in general outperform base-level classifiers and that open issue is how such a system would perform when given data plurality voting, which combines multiple base-level classifiers, from an unknown orientation. In this paper we propose a method shows the best results. The authors also showed that out of the that only partially limits the device orientation, in a way that the used features, energy is the least significant one, and that there is device can be rotated only around one axis, but for an arbitrary no significant change in accuracy when this feature is avoided. angle. This arbitrary rotation practically creates an infinite number of possible orientations, in contrast of previous approaches, which Kwapisz et al. in [4] tried to recognize activities by using data are all limited to a certain number of orientations. To achieve this, from a single acceleration sensor, but they used data from an a calibration phase which precedes activity recognition is acceleration sensor embedded into a standard mobile phone. introduced. These accelerometers typically detect acceleration up to ±2G along three axes. Their research methodology follows the one in [2, 3]. The authors collected data from 29 subjects, extracted 6 3. ACCELEROMETER DATA basic features and tested 3 classifiers, where multilayer COLLECTION perceptrons showed the best result, recognizing activities with an As a test device, a smartphone Samsung I9001 Galaxy S Plus accuracy of 91.7%. The authors showed that activity recognition which runs on Android operating system version 2.3.5 is used. can be performed successfully by using acceleration data from a The accelerometer embedded in this phone detects acceleration up mobile phone. to ±2G. Data from the accelerometer has three attributes: The unifying fact for papers [2 - 4], no matter if one or more acceleration along X, Y and Z axis, represented by floating point accelerometers are used, is that the position and the orientation of values. Sampling rate for the accelerometer was set to the accelerometer is fixed while performing all of the examined SENSOR_DELAY_FASTEST to achieve the highest possible activities. This fact can be probably expected in case of accuracy. specialized devices as in [2, 3]. In case of using a standard mobile An application for recording data from the accelerometer has been phone as in [4], the method puts strains on how someone carries a developed. Data was collected by several test users. The recording 2 Proceedings IMMoA’13 16 http://www.dbis.rwth-aachen.de/IMMoA2013/ process is as follows: while standing still, test user selects the applied in a real-time locally on a device. For the testing we used activity he is going to perform and starts the recording. After that 10-fold cross validation, and the results are shown in Table 1. All the user has ten seconds to place the phone in the pocket in the of the tested classifiers showed excellent results in recognizing desired orientation. After ten seconds a beep sound is played and activities, which is consistent with previous work [5]. The results for two seconds the gravity vector is extracted from accelerometer are slightly better than in [5] which can be probably contributed data. After two seconds another beep sound is played and an to a specific data set and the fact that the data was collected by a average value for the gravity vector is saved to a file. To extract single test user. the gravity vector from accelerometer data a simple low-pass filter is used. The user can then start to perform the specified activity. Table 1. Classifier accuracy – Mixed orientation data Another 2 seconds after the second beep, the application starts to approach record acceleration data to another file. After finishing with the Classifier Accuracy (in %) activity the user stops the recording. C4.5 Decision Tree 98.8 Data was collected while performing 6 different activities: Naïve Bayes 99.5  Standing K-nearest neighbors 99.8  Walking  Running  Walking up stairs 5. A DATA REORIENTATION  Walking down stairs PREPROCESSING APPROACH  Sitting. The second approach to the free orientation problem is based on For each activity data was collected for the default orientation and building a classifier from data collected in the default orientation. for 3-4 other orientations, depending on the activity. Some of the In the classifying phase, transformation of data collected in non-default orientations matched between activities and some did various orientations into the default coordinate system is not. To minimize mislabeling a portion of data was removed from performed, prior to the feature extraction and classification. the beginning and the end of each recording. The classifiers for testing are built in the same way as in the mixed orientation data approach, but now only data collected 4. A MIXED ORIENTATION DATA from the default orientation is used. To get the most precise APPROACH results, it is important to test the classifier with data from all The first approach to the free orientation problem we test is available orientations, which includes also data from the default building a classifier from data collected from all orientations, very orientation. For this reason, a portion of data from the default similar to [5]. In this approach, the next step after data collection orientation was omitted in classifier building, and was used later is feature extraction. The features were extracted from in classifier testing. In this way we avoid overfitting, which can accelerometer data using a window size of 512 samples with 256 happen when the same data is used for training and testing. No samples overlapping between consecutive windows. Three data transformation was done on data used for building a features were extracted from each of the three axes, giving a total classifier. of nine attributes for building a classifier. The features extracted As previously mentioned, in this paper we assume that the phone were: can be rotated only around the Z axis, and consequently, we  Mean assume that there is no change in the acceleration along the Z axis  Standard deviation when performing some activity in the default and non-default  Correlation. position. This means that we do not need to transform the Z We specified these features, that are calculated using data in time coordinate, just X and Y coordinates. To achieve that, we use a domain, because we apply the activity recognition system in real- rotation matrix for rotation around the Z axis for an angle θ. The time locally on a device. For this reason, the features should be rotation matrix is given in (1). To calculate angle θ we use the relatively simple to compute, to reduce power consumption and information about gravity vectors. processor load. The selected features do not require signal representation in the frequency domain, and thus can be computed cos( )  sin( ) 0 relatively fast. Also, the mean is used in the standard deviation R   sin( ) cos( ) 0 (1)   calculation, and the mean and the standard deviation are used in  0 0 1 correlation calculation, which further increases computation speed. By using the gravity vectors from all of the recordings in the Extracted features were used to train and test 3 classifiers default orientation, we computed an average gravity vector for the available in the WEKA Data Mining Toolkit [7], which are default orientation (gravity vector is defined as data from the commonly used in activity recognition [2 – 4, 6]: accelerometer, by three attributes: X, Y and Z). Since the phone’s  C4.5 decision tree screen is facing the user, the Z axis is practically parallel to the ground, so we take into account only the X and Y components of  Naïve Bayes the vector.  K-nearest neighbors. We are mainly interested in the performance of the decision tree classifier, since it requires the least amount of computation in the classification phase, which is important when the system is 3 Proceedings IMMoA’13 17 http://www.dbis.rwth-aachen.de/IMMoA2013/ It can be seen that a lot of instances that represent sitting are classified as standing, which is not intuitive, because these two activities should be easy to distinguish. This is a consequence of how WEKA generates the decision tree. When we look at the generated decision tree shown in Figure 3, we can see that sitting and standing are distinguished by mean value along the Z axis (MeanZ). WEKA makes the split on value -9.804189 which is the maximum value for MeanZ for sitting. It can be expected that the maximum value for MeanZ will vary for different recordings, since it can’t be expected from someone to sit in exactly the same way every time. Since the minimum value for MeanZ for standing is -1.07289 the split should be done on the value -5.4385, which is halfway between the maximum for sitting and the minimum for Figure 1. The accelerometer data collected while walking standing. This would generate a much more robust tree with higher accuracy. Such decision tree would correctly classify all of In the classifying phase, we firstly calculate the difference the sitting instances, previously misclassified as standing. between the angle of the average gravity vector in the default Consequently, decision tree accuracy increases to 92%. orientation and the angle of the gravity vector for the current orientation, and this difference is angle θ we need to transform accelerometer data into the default coordinate system. In the next step, each sample from the accelerometer is transformed into the default coordinate system using the rotation matrix given in (1). Figure 1 shows data from the accelerometer while walking. The vertical axis represents the acceleration in m/s2. Samples from the acceleration sensor are represented along the horizontal axis. Samples 1-250 represent data when the phone is in the default orientation, samples 251-500 represent data when the phone is in a non-default orientation and samples 501-750 represent the same Figure 3. Generated decision tree data as samples 251-500, but transformed into the default To demonstrate the benefits of this approach, the same classifier coordinate system. Transformed data is then used to extract was tested with the same test data, but this time no data features in the same way as in the mixed orientation data transformation was performed prior to classification. Based on the approach. For testing we used data from non-default orientations, results shown in Table 3, it can be concluded that a classifier built and data from default orientation. Data from default orientation using data from only one orientation, cannot classify instances was treated the same way as data from non-default orientations, from other orientations with a high success rate. With the and was transformed accordingly to its gravity vector. We tested reorientation preprocessing included, the classification accuracy the same 3 classifiers as in the mixed orientation data approach, results (shown in Table 2) increase significantly, which and the results are shown in Table 2. demonstrates the advantage of this approach, compared to the one Table 2. Classifier accuracy – Data reorientation assuming a fixed orientation at all times. preprocessing approach Table 3. Classifier accuracy – Single orientation classifiers Classifier Accuracy (in %) without reorientation C4.5 Decision Tree 86.5 Classifier Accuracy (in %) Naïve Bayes 91.5 C4.5 Decision Tree 54.4 K-nearest neighbors 95.0 Naïve Bayes 48.5 K-nearest neighbors 58.8 The results obtained are lower than in the mixed orientation data approach, but are still above 90% threshold, except for the decision tree classifier. For this reason we analyze the decision To compare the data reorientation preprocessing approach against tree classifier further. The confusion matrix for the decision tree the mixed orientation data approach, the classifier built with the classifier is shown in Figure 2. mixed orientation data approach was retested with a data set consisting of classifier training data, as well as data including orientations that were not used in classifier training. Accuracy of the mixed orientation data approach, when handling data including unknown orientations, can be analyzed in this manner. Evaluation results are shown in Table 4 and are comparable to the results achieved with the data reorientation preprocessing approach. It can be concluded that the mixed orientation data approach can also handle data from previously unknown orientations, but with a decrease in accuracy compared to the data Figure 2. Confusion matrix for the decision tree classifier reorientation preprocessing approach. 4 Proceedings IMMoA’13 18 http://www.dbis.rwth-aachen.de/IMMoA2013/ Table 4. Classifier accuracy - Mixed orientation data approach notice three color groups, where each one, looking from the left to with unknown orientations the right, represents calculation of mean, standard deviation and correlation. The box marked with the number 2 represents data Classifier Accuracy (in %) transformation. We can see how data transformation is performed C4.5 Decision Tree 85.0 whenever a new sample is read from the accelerometer and that Naïve Bayes 83.8 feature extraction is performed only when the window shifts for K-nearest neighbors 88.5 the specified number of samples. The Figure 5 shows the timeline again but this time in more 6. EVALUATION OF MOBILE CPU LOAD details. Similar to Figure 4, the box marked with the number 1 Using results from the previous test we built an application for represents mean calculation (green color on the timeline), and the activity recognition in real-time locally on the device. The box marked with the number 2 represents data transformation. It application implements the data reorientation preprocessing can be seen that data transformation requires a small portion of approach and uses a prepared decision tree as a classifier. To test processor time, compared to processor time required to calculate how much the application and data transformation phase just one feature, but is performed more often. participate in the processor load we used DDMS (Dalvik Debug Monitor Service). The entry point of the application is the onSensorChanged function, which is called whenever a new sample from the We ran DDMS for thirty seconds while the application was active accelerometer is read. This function encapsulates all of the on the device. The results are shown in Figure 4. The first line of application processing and participates in the processor load with the figure represents the main thread of the application in which 22.9%. Also, the function transformData, which encapsulates all all of the application processing is done. Different methods are of the data transformation, participates in the processor load with represented with different colors on the timeline. This figure 1.3%. It can be concluded that although data transformation is focuses on the period between two feature extractions. The boxes performed more often than feature extraction, it doesn't increase marked with the number 1 represent feature calculation. We processor load significantly. Figure 4. Result of the DDMS tool applied on the activity recognition application Figure 5. Result of the DDMS tool in more detail 5 Proceedings IMMoA’13 19 http://www.dbis.rwth-aachen.de/IMMoA2013/ Figure 6. Participation of individual functions in processor load 7. CONCLUSION AND FUTURE WORK 8. ACKNOWLEDGMENTS In this paper a method for orientation independent activity This paper was realized as a part of the project "Studying climate recognition from accelerometer data is described. An change and its influence on the environment: impacts, adaptation accelerometer embedded in a mobile phone was used. Orientation and mitigation" (43007) financed by the Ministry of Education, of the phone was partially fixed in such a way, that the phone Science and Technological Development of the Republic of could be rotated only around one axis, but the angle of orientation Serbia within the framework of integrated and interdisciplinary was arbitrary. To determine the angle of rotation a calibration research for the period 2011-2014. phase was introduced, in which the user has to stand still for a couple of seconds with the phone placed in the desired orientation. In this period the gravity vector is extracted and the 9. REFERENCES [1] J. Lester, T. Choudhury, G. Booriello. A Practical Approach difference between the angle of that gravity vector and the angle to Recognizing Physical Activities. Proceedings of the 4th of the average gravity vector in the default orientation is International Conference on Pervasive Computing, 2006, pp. calculated. This difference is the angle of rotation of the phone. 1-16. After that the user can start to perform activities freely. The data from the accelerometer is transformed into the default coordinate [2] L. Bao, S. S. Intille. Activity recognition from user-annotated system, the features are extracted and the activity recognition is acceleration data. Proceceedings of the 2nd International performed. Conference on Pervasive Computing, 2004, pp. 1-17. This method showed slightly reduced accuracy compared to the [3] N. Ravi, N. Dandekar, P. Mysore, M. L. Littman. Activity method when a classifier is built from data collected from various recognition from accelerometer data. Proceedings of the orientations, when data from the predefined orientations only is Seventeenth Conference on Innovative Applications of considered, but the results are still above the threshold (accuracy Artificial Intelligence (IAAI-05), 2005, pp. 1541-1546. above 90%). When data from not predefined orientations is considered as well, the proposed method demonstrates increased [4] J. R. Kwapisz, G. M. Weiss, S. A. Moore. Activity accuracy. The most significant advantage of this method is that it Recognition using Cell Phone Accelerometers. ACM requires data collection from only one orientation, so less data is SIGKDD Explorations Newsletter, 2010, vol. 12, issue 2, pp. required for training. Also it makes no assumption on the 74-82. orientation in the classifying phase; there are no predefined [5] L. Sun, D. Zhang, B. Li, B. Guo, S. Li. Activity Recognition orientations, so the system will work with data from any on an Accelerometer Embedded Mobile Phone with Varying orientation. The drawback is the existence of the calibration Positions and Orientations. Proceedings of the 7th phase, so the process in not fully transparent. This probably International Conference on Ubiquitous Intelligence and makes this method unusable in some areas of application, like in Computing, 2010, pp. 548-562. elderly care for example, but in others, like in fitness monitoring, we believe that is acceptable because the calibration phase is very [6] S. Thiemjarus. A Device-Orientation Independent Method short and requires very little effort from the user. In return the for Activity Recognition. International Conference on Body user can place the phone in any orientation. Sensor Networks, 2010, pp. 19 - 23. In this paper the phone orientation is assumed to be partially [7] I. H. Witte, E. Frank. Data Mining: Practical Machine fixed, which still limits the user to a certain degree. Demonstrated Learning Tools and Techniques, 3rd ed. Morgan Kaufmann, method represents only an intermediate step in development of a 2011 method which would acquire a full three-dimensional orientation in the calibration phase and impose no limits in the phone’s orientation. 6 Proceedings IMMoA’13 20 http://www.dbis.rwth-aachen.de/IMMoA2013/