This paper demonstrates a method for activity recognition partially resilient on mobile device orientation, by using data from a mobile phone embedded accelerometer. This method is partially resilient on mobile device orientation, in such a way that a mobile device can be rotated around only one axis for an arbitrary angle. The classifier for activity recognition is built using data from one default orientation. This method introduces a calibration phase in which the phone's orientation is determined. After that, accelerometer data is transformed into the default coordinate system and further processed. The solution is compared with the method that built a classifier using data from multiple orientations. Three classifiers were tested and a high accuracy of around 90% was achieved for all of them.
The greatest possibilities for application of activity recognition
systems lay in the healthcare domain. For example, such systems
can be used for elderly care support or for long-term health/fitness
monitoring [
In recent years a lot of work has been done on activity recognition from accelerometer data. Since an accelerometer is a standard part of modern mobile devices (like mobile phones and tablets), they can be used in activity recognition. An advantage in using these devices is that they are already commonly used by a lot of people that would not have to wear an additional device to perform activity recognition, which greatly increases the acceptance of such a system.
This paper focuses on recognizing activities from accelerometer data processed at a mobile phone. Activity recognition is formulated as a classification problem. This paper considers the healthcare domain for activity recognition system application. For this reason it is important to recognize physical activities, such as: walking, running, walking up/down stairs etc. Examples of possible users in this domain are the elderly and persons with certain disabilities. It can be assumed that such users keep their mobile phone relatively (but not completely) fixed. In the paper it is assumed that the phone position is fixed, but that the phone orientation is only partially fixed. The orientation is partially fixed in such a way that the axis that is perpendicular to the phone’s screen is parallel to the ground and the phone can be rotated freely around that axis. For experiments in this paper, the mobile phone was worn in the right front pants pocket (as one of the places where people usually carry mobile phones) and the screen of the phone was facing the user. The orientation where the bottom side of the phone is facing the ground is considered the default orientation. First, three classifiers were built using data from multiple orientations. These classifiers were tested using data from the same orientations. Then, classifiers were built again, but using data from the default orientation only. These new classifiers were tested using data from multiple orientations, transformed into the default coordinate system prior to testing, according to the method proposed in this paper. After that, classifier performances from these two tests were compared.
Using results from previously mentioned tests, we created an application for activity recognition that runs on a mobile device in real time and tested the impact that data transformation has on the performance of such an application, specifically in terms of processor load. The processor load is important because if such application is to be accepted by users, it must not significantly decrease battery life, or the performance of the device in everyday tasks. The proposed method represents only an intermediate step in the development of a method for activity recognition with no restrictions in mobile device orientation.
The rest of the paper is structured as follows: section 2 provides an overview of related work on activity recognition. Section 3 describes the process of accelerometer data collection. In sections 4 and 5, mixed orientation data and data reorientation processing approach, for activity recognition resilient on mobile device orientation, are presented and evaluated. Section 6 presents an evaluation of how much the activity recognition application and data transformation phase participate in the processor load. Section 7 gives the conclusions about the paper and outlines plans for future work.
In recent years there has been a lot of research related to
recognizing activities from accelerometer data. In [
Ravi et al. in [
The unifying fact for papers [
Sun et al. in [
Thiemjarus in [
A possible problem with the approach in [
As a test device, a smartphone Samsung I9001 Galaxy S Plus which runs on Android operating system version 2.3.5 is used. The accelerometer embedded in this phone detects acceleration up to ±2G. Data from the accelerometer has three attributes: acceleration along X, Y and Z axis, represented by floating point values. Sampling rate for the accelerometer was set to SENSOR_DELAY_FASTEST to achieve the highest possible accuracy.
An application for recording data from the accelerometer has been developed. Data was collected by several test users. The recording process is as follows: while standing still, test user selects the activity he is going to perform and starts the recording. After that the user has ten seconds to place the phone in the pocket in the desired orientation. After ten seconds a beep sound is played and for two seconds the gravity vector is extracted from accelerometer data. After two seconds another beep sound is played and an average value for the gravity vector is saved to a file. To extract the gravity vector from accelerometer data a simple low-pass filter is used. The user can then start to perform the specified activity. Another 2 seconds after the second beep, the application starts to record acceleration data to another file. After finishing with the activity the user stops the recording.
Data was collected while performing 6 different activities: Standing Walking Running Walking up stairs Walking down stairs Sitting.
For each activity data was collected for the default orientation and for 3-4 other orientations, depending on the activity. Some of the non-default orientations matched between activities and some did not. To minimize mislabeling a portion of data was removed from the beginning and the end of each recording.
The first approach to the free orientation problem we test is
building a classifier from data collected from all orientations, very
similar to [
We specified these features, that are calculated using data in time domain, because we apply the activity recognition system in realtime locally on a device. For this reason, the features should be relatively simple to compute, to reduce power consumption and processor load. The selected features do not require signal representation in the frequency domain, and thus can be computed relatively fast. Also, the mean is used in the standard deviation calculation, and the mean and the standard deviation are used in correlation calculation, which further increases computation speed.
Extracted features were used to train and test 3 classifiers
available in the WEKA Data Mining Toolkit [
We are mainly interested in the performance of the decision tree
classifier, since it requires the least amount of computation in the
classification phase, which is important when the system is
applied in a real-time locally on a device. For the testing we used
10-fold cross validation, and the results are shown in Table 1. All
of the tested classifiers showed excellent results in recognizing
activities, which is consistent with previous work [
The second approach to the free orientation problem is based on building a classifier from data collected in the default orientation. In the classifying phase, transformation of data collected in various orientations into the default coordinate system is performed, prior to the feature extraction and classification. The classifiers for testing are built in the same way as in the mixed orientation data approach, but now only data collected from the default orientation is used. To get the most precise results, it is important to test the classifier with data from all available orientations, which includes also data from the default orientation. For this reason, a portion of data from the default orientation was omitted in classifier building, and was used later in classifier testing. In this way we avoid overfitting, which can happen when the same data is used for training and testing. No data transformation was done on data used for building a classifier.
As previously mentioned, in this paper we assume that the phone can be rotated only around the Z axis, and consequently, we assume that there is no change in the acceleration along the Z axis when performing some activity in the default and non-default position. This means that we do not need to transform the Z coordinate, just X and Y coordinates. To achieve that, we use a rotation matrix for rotation around the Z axis for an angle θ. The rotation matrix is given in (1). To calculate angle θ we use the information about gravity vectors.
cos( ) R sin( ) 0 sin( ) 0 cos( ) 0 0 1 (1) By using the gravity vectors from all of the recordings in the default orientation, we computed an average gravity vector for the default orientation (gravity vector is defined as data from the accelerometer, by three attributes: X, Y and Z). Since the phone’s screen is facing the user, the Z axis is practically parallel to the ground, so we take into account only the X and Y components of the vector. In the classifying phase, we firstly calculate the difference between the angle of the average gravity vector in the default orientation and the angle of the gravity vector for the current orientation, and this difference is angle θ we need to transform accelerometer data into the default coordinate system. In the next step, each sample from the accelerometer is transformed into the default coordinate system using the rotation matrix given in (1). Figure 1 shows data from the accelerometer while walking. The vertical axis represents the acceleration in m/s2. Samples from the acceleration sensor are represented along the horizontal axis. Samples 1-250 represent data when the phone is in the default orientation, samples 251-500 represent data when the phone is in a non-default orientation and samples 501-750 represent the same data as samples 251-500, but transformed into the default coordinate system. Transformed data is then used to extract features in the same way as in the mixed orientation data approach. For testing we used data from non-default orientations, and data from default orientation. Data from default orientation was treated the same way as data from non-default orientations, and was transformed accordingly to its gravity vector. We tested the same 3 classifiers as in the mixed orientation data approach, and the results are shown in Table 2. The results obtained are lower than in the mixed orientation data approach, but are still above 90% threshold, except for the decision tree classifier. For this reason we analyze the decision tree classifier further. The confusion matrix for the decision tree classifier is shown in Figure 2. It can be seen that a lot of instances that represent sitting are classified as standing, which is not intuitive, because these two activities should be easy to distinguish. This is a consequence of how WEKA generates the decision tree. When we look at the generated decision tree shown in Figure 3, we can see that sitting and standing are distinguished by mean value along the Z axis (MeanZ). WEKA makes the split on value -9.804189 which is the maximum value for MeanZ for sitting. It can be expected that the maximum value for MeanZ will vary for different recordings, since it can’t be expected from someone to sit in exactly the same way every time. Since the minimum value for MeanZ for standing is -1.07289 the split should be done on the value -5.4385, which is halfway between the maximum for sitting and the minimum for standing. This would generate a much more robust tree with higher accuracy. Such decision tree would correctly classify all of the sitting instances, previously misclassified as standing. Consequently, decision tree accuracy increases to 92%. To demonstrate the benefits of this approach, the same classifier was tested with the same test data, but this time no data transformation was performed prior to classification. Based on the results shown in Table 3, it can be concluded that a classifier built using data from only one orientation, cannot classify instances from other orientations with a high success rate. With the reorientation preprocessing included, the classification accuracy results (shown in Table 2) increase significantly, which demonstrates the advantage of this approach, compared to the one assuming a fixed orientation at all times. To compare the data reorientation preprocessing approach against the mixed orientation data approach, the classifier built with the mixed orientation data approach was retested with a data set consisting of classifier training data, as well as data including orientations that were not used in classifier training. Accuracy of the mixed orientation data approach, when handling data including unknown orientations, can be analyzed in this manner. Evaluation results are shown in Table 4 and are comparable to the results achieved with the data reorientation preprocessing approach. It can be concluded that the mixed orientation data approach can also handle data from previously unknown orientations, but with a decrease in accuracy compared to the data reorientation preprocessing approach.
Using results from the previous test we built an application for activity recognition in real-time locally on the device. The application implements the data reorientation preprocessing approach and uses a prepared decision tree as a classifier. To test how much the application and data transformation phase participate in the processor load we used DDMS (Dalvik Debug Monitor Service).
We ran DDMS for thirty seconds while the application was active on the device. The results are shown in Figure 4. The first line of the figure represents the main thread of the application in which all of the application processing is done. Different methods are represented with different colors on the timeline. This figure focuses on the period between two feature extractions. The boxes marked with the number 1 represent feature calculation. We notice three color groups, where each one, looking from the left to the right, represents calculation of mean, standard deviation and correlation. The box marked with the number 2 represents data transformation. We can see how data transformation is performed whenever a new sample is read from the accelerometer and that feature extraction is performed only when the window shifts for the specified number of samples.
The Figure 5 shows the timeline again but this time in more details. Similar to Figure 4, the box marked with the number 1 represents mean calculation (green color on the timeline), and the box marked with the number 2 represents data transformation. It can be seen that data transformation requires a small portion of processor time, compared to processor time required to calculate just one feature, but is performed more often.
The entry point of the application is the onSensorChanged function, which is called whenever a new sample from the accelerometer is read. This function encapsulates all of the application processing and participates in the processor load with 22.9%. Also, the function transformData, which encapsulates all of the data transformation, participates in the processor load with 1.3%. It can be concluded that although data transformation is performed more often than feature extraction, it doesn't increase processor load significantly.
In this paper a method for orientation independent activity recognition from accelerometer data is described. An accelerometer embedded in a mobile phone was used. Orientation of the phone was partially fixed in such a way, that the phone could be rotated only around one axis, but the angle of orientation was arbitrary. To determine the angle of rotation a calibration phase was introduced, in which the user has to stand still for a couple of seconds with the phone placed in the desired orientation. In this period the gravity vector is extracted and the difference between the angle of that gravity vector and the angle of the average gravity vector in the default orientation is calculated. This difference is the angle of rotation of the phone. After that the user can start to perform activities freely. The data from the accelerometer is transformed into the default coordinate system, the features are extracted and the activity recognition is performed.
This method showed slightly reduced accuracy compared to the method when a classifier is built from data collected from various orientations, when data from the predefined orientations only is considered, but the results are still above the threshold (accuracy above 90%). When data from not predefined orientations is considered as well, the proposed method demonstrates increased accuracy. The most significant advantage of this method is that it requires data collection from only one orientation, so less data is required for training. Also it makes no assumption on the orientation in the classifying phase; there are no predefined orientations, so the system will work with data from any orientation. The drawback is the existence of the calibration phase, so the process in not fully transparent. This probably makes this method unusable in some areas of application, like in elderly care for example, but in others, like in fitness monitoring, we believe that is acceptable because the calibration phase is very short and requires very little effort from the user. In return the user can place the phone in any orientation.
In this paper the phone orientation is assumed to be partially fixed, which still limits the user to a certain degree. Demonstrated method represents only an intermediate step in development of a method which would acquire a full three-dimensional orientation in the calibration phase and impose no limits in the phone’s orientation.
This paper was realized as a part of the project "Studying climate change and its influence on the environment: impacts, adaptation and mitigation" (43007) financed by the Ministry of Education, Science and Technological Development of the Republic of Serbia within the framework of integrated and interdisciplinary research for the period 2011-2014.