=Paper=
{{Paper
|id=None
|storemode=property
|title=A Method for Activity Recognition Partially Resilient on Mobile Device Orientation
|pdfUrl=https://ceur-ws.org/Vol-1075/02.pdf
|volume=Vol-1075
|dblpUrl=https://dblp.org/rec/conf/immoa/JajacPS13
}}
==A Method for Activity Recognition Partially Resilient on Mobile Device Orientation==
https://ceur-ws.org/Vol-1075/02.pdf
A Method for Activity Recognition Partially Resilient on
Mobile Device Orientation
Nikola Jajac, Bratislav Predic, Dragan Stojanovic
University of Nis, Faculty of Electronic Engineering
18000 Nis, Serbia
{nikola.jajac, bratislav.predic, dragan.stojanovic}@elfak.ni.ac.rs
ABSTRACT system for recognizing activities could help reduce errors that
This paper demonstrates a method for activity recognition arise from previously mentioned methods. Also, such system
partially resilient on mobile device orientation, by using data from enables users to go about their daily routines, while the data
a mobile phone embedded accelerometer. This method is partially collection and processing are done in the background, and do not
resilient on mobile device orientation, in such a way that a mobile interfere with their current activities.
device can be rotated around only one axis for an arbitrary angle. In recent years a lot of work has been done on activity recognition
The classifier for activity recognition is built using data from one from accelerometer data. Since an accelerometer is a standard part
default orientation. This method introduces a calibration phase in of modern mobile devices (like mobile phones and tablets), they
which the phone’s orientation is determined. After that, can be used in activity recognition. An advantage in using these
accelerometer data is transformed into the default coordinate devices is that they are already commonly used by a lot of people
system and further processed. The solution is compared with the that would not have to wear an additional device to perform
method that built a classifier using data from multiple activity recognition, which greatly increases the acceptance of
orientations. Three classifiers were tested and a high accuracy of such a system.
around 90% was achieved for all of them.
This paper focuses on recognizing activities from accelerometer
data processed at a mobile phone. Activity recognition is
1. INTRODUCTION formulated as a classification problem. This paper considers the
The advance of low-cost and low-power sensors has led to their healthcare domain for activity recognition system application. For
massive integration into modern mobile devices that have become this reason it is important to recognize physical activities, such as:
powerful sensing platforms. By using data from sensors like walking, running, walking up/down stairs etc. Examples of
accelerometers, gyroscopes, digital compasses, light sensors etc. it possible users in this domain are the elderly and persons with
is possible to describe the context of a mobile device user in much certain disabilities. It can be assumed that such users keep their
more detail than by using only location data. A richer description mobile phone relatively (but not completely) fixed. In the paper it
of a user context provides better adaptation of mobile content, is assumed that the phone position is fixed, but that the phone
services and resources, enabling the user to stay focused on the orientation is only partially fixed. The orientation is partially fixed
task at hand. in such a way that the axis that is perpendicular to the phone’s
An activity of the user represents an important aspect of a context, screen is parallel to the ground and the phone can be rotated freely
because it directly impacts user’s ability to interact with the around that axis. For experiments in this paper, the mobile phone
mobile device and applications. By employing the information was worn in the right front pants pocket (as one of the places
about the current activity, the mobile device can adapt its where people usually carry mobile phones) and the screen of the
interfaces, filter the content it provides, or perform a specified phone was facing the user. The orientation where the bottom side
action, to support the user in the best possible way. of the phone is facing the ground is considered the default
orientation. First, three classifiers were built using data from
The greatest possibilities for application of activity recognition multiple orientations. These classifiers were tested using data
systems lay in the healthcare domain. For example, such systems from the same orientations. Then, classifiers were built again, but
can be used for elderly care support or for long-term health/fitness using data from the default orientation only. These new classifiers
monitoring [1]. Current methods for tracking activities, like were tested using data from multiple orientations, transformed
paying a trained observer or relying on self-reporting are time and into the default coordinate system prior to testing, according to
resource consuming tasks, and are error prone. An automatic the method proposed in this paper. After that, classifier
performances from these two tests were compared.
Using results from previously mentioned tests, we created an
application for activity recognition that runs on a mobile device in
real time and tested the impact that data transformation has on the
performance of such an application, specifically in terms of
processor load. The processor load is important because if such
application is to be accepted by users, it must not significantly
decrease battery life, or the performance of the device in everyday
tasks.
1
Proceedings IMMoA’13 15 http://www.dbis.rwth-aachen.de/IMMoA2013/
The proposed method represents only an intermediate step in the mobile device, which could decrease acceptability of such a
development of a method for activity recognition with no system.
restrictions in mobile device orientation.
Sun et al. in [5] tried to recognize activities by using acceleration
The rest of the paper is structured as follows: section 2 provides data from a mobile phone, in a setting where the position and the
an overview of related work on activity recognition. Section 3 orientation of the phone vary. They restrict their hypothesis space
describes the process of accelerometer data collection. In sections to 6 possible positions (6 pockets) and 4 orientations of the
4 and 5, mixed orientation data and data reorientation processing mobile phone. The data from all position and orientation
approach, for activity recognition resilient on mobile device combinations were collected. The authors added acceleration
orientation, are presented and evaluated. Section 6 presents an magnitude at each sample, as an additional sensor reading
evaluation of how much the activity recognition application and dimension. By using collected data, several features were
data transformation phase participate in the processor load. calculated: mean, variance, correlation, FFT energy and
Section 7 gives the conclusions about the paper and outlines plans frequency-domain entropy. Calculated features were used to train
for future work. and test SVM (Support Vector Machine) models. Generated SVM
model recognizes activities with an accuracy of 93.1% throughout
all tested positions and orientations.
2. RELATED WORK
In recent years there has been a lot of research related to Thiemjarus in [6] applied a different approach. Accelerometer was
recognizing activities from accelerometer data. In [2] authors used mounted on a belt-clip device which was worn by test subjects in
data from 5 biaxial accelerometers worn simultaneously on a fixed position on a body, but which could be mounted in 4
different parts of the body. Used accelerometers could detect different orientations. Data was collected by 13 subjects that
acceleration up to ±10G. Accelerometers were mounted onto performed a routine comprised of 6 activities. The first step in
hoarder boards and firmly attached to different body parts. Data data analysis was device orientation detection. The orientation
was collected from 20 subjects performing various everyday tasks detection was also formulated as a classification problem. The
without researcher supervision. The following features were features used for orientation detection were mean along Y and Z
computed on sliding windows of accelerometer data: mean, axis. Orientation detection was performed for an activity routine
energy, frequency-domain entropy and correlation. A number of performance, which contains approximately 5 seconds of data for
classifiers were trained and tested with the calculated data, where each tested activity, while activity recognition was window-based.
decision trees showed the best result, recognizing activities with The second step was signal transformation using the appropriate
an accuracy of 84%. transformation matrix, and the third step was activity recognition
itself. Author achieved a subject-independent classification
Ravi et al. in [3] attempted to perform activity recognition using a
accuracy of 90.9%.
single triaxial accelerometer worn near the pelvic region. Data
was collected by 2 subjects performing 8 different activities. A possible problem with the approach in [6] is that orientation
Similarly to [2] the features were computed using the sliding detection is done on a data set that includes information about all
windows technique. Four features were extracted: mean, standard tested activities. The open issue is how this approach can be
deviation, energy and correlation. Extracted features were used to applied in a real world scenario, when the user does not perform a
train and test 5 base-level classifiers, and in addition to that, 5 specified activity routine. A unifying fact for [5, 6] is that the
meta-level classifiers. Authors concluded that meta-level hypothesis space is limited to a number of orientations. Again, the
classifiers in general outperform base-level classifiers and that open issue is how such a system would perform when given data
plurality voting, which combines multiple base-level classifiers, from an unknown orientation. In this paper we propose a method
shows the best results. The authors also showed that out of the that only partially limits the device orientation, in a way that the
used features, energy is the least significant one, and that there is device can be rotated only around one axis, but for an arbitrary
no significant change in accuracy when this feature is avoided. angle. This arbitrary rotation practically creates an infinite number
of possible orientations, in contrast of previous approaches, which
Kwapisz et al. in [4] tried to recognize activities by using data
are all limited to a certain number of orientations. To achieve this,
from a single acceleration sensor, but they used data from an
a calibration phase which precedes activity recognition is
acceleration sensor embedded into a standard mobile phone.
introduced.
These accelerometers typically detect acceleration up to ±2G
along three axes. Their research methodology follows the one in
[2, 3]. The authors collected data from 29 subjects, extracted 6 3. ACCELEROMETER DATA
basic features and tested 3 classifiers, where multilayer COLLECTION
perceptrons showed the best result, recognizing activities with an As a test device, a smartphone Samsung I9001 Galaxy S Plus
accuracy of 91.7%. The authors showed that activity recognition which runs on Android operating system version 2.3.5 is used.
can be performed successfully by using acceleration data from a The accelerometer embedded in this phone detects acceleration up
mobile phone. to ±2G. Data from the accelerometer has three attributes:
The unifying fact for papers [2 - 4], no matter if one or more acceleration along X, Y and Z axis, represented by floating point
accelerometers are used, is that the position and the orientation of values. Sampling rate for the accelerometer was set to
the accelerometer is fixed while performing all of the examined SENSOR_DELAY_FASTEST to achieve the highest possible
activities. This fact can be probably expected in case of accuracy.
specialized devices as in [2, 3]. In case of using a standard mobile An application for recording data from the accelerometer has been
phone as in [4], the method puts strains on how someone carries a developed. Data was collected by several test users. The recording
2
Proceedings IMMoA’13 16 http://www.dbis.rwth-aachen.de/IMMoA2013/
process is as follows: while standing still, test user selects the applied in a real-time locally on a device. For the testing we used
activity he is going to perform and starts the recording. After that 10-fold cross validation, and the results are shown in Table 1. All
the user has ten seconds to place the phone in the pocket in the of the tested classifiers showed excellent results in recognizing
desired orientation. After ten seconds a beep sound is played and activities, which is consistent with previous work [5]. The results
for two seconds the gravity vector is extracted from accelerometer are slightly better than in [5] which can be probably contributed
data. After two seconds another beep sound is played and an to a specific data set and the fact that the data was collected by a
average value for the gravity vector is saved to a file. To extract single test user.
the gravity vector from accelerometer data a simple low-pass filter
is used. The user can then start to perform the specified activity. Table 1. Classifier accuracy – Mixed orientation data
Another 2 seconds after the second beep, the application starts to approach
record acceleration data to another file. After finishing with the Classifier Accuracy (in %)
activity the user stops the recording. C4.5 Decision Tree 98.8
Data was collected while performing 6 different activities: Naïve Bayes 99.5
Standing K-nearest neighbors 99.8
Walking
Running
Walking up stairs 5. A DATA REORIENTATION
Walking down stairs PREPROCESSING APPROACH
Sitting. The second approach to the free orientation problem is based on
For each activity data was collected for the default orientation and building a classifier from data collected in the default orientation.
for 3-4 other orientations, depending on the activity. Some of the In the classifying phase, transformation of data collected in
non-default orientations matched between activities and some did various orientations into the default coordinate system is
not. To minimize mislabeling a portion of data was removed from performed, prior to the feature extraction and classification.
the beginning and the end of each recording.
The classifiers for testing are built in the same way as in the
mixed orientation data approach, but now only data collected
4. A MIXED ORIENTATION DATA from the default orientation is used. To get the most precise
APPROACH results, it is important to test the classifier with data from all
The first approach to the free orientation problem we test is available orientations, which includes also data from the default
building a classifier from data collected from all orientations, very orientation. For this reason, a portion of data from the default
similar to [5]. In this approach, the next step after data collection orientation was omitted in classifier building, and was used later
is feature extraction. The features were extracted from in classifier testing. In this way we avoid overfitting, which can
accelerometer data using a window size of 512 samples with 256 happen when the same data is used for training and testing. No
samples overlapping between consecutive windows. Three data transformation was done on data used for building a
features were extracted from each of the three axes, giving a total classifier.
of nine attributes for building a classifier. The features extracted
As previously mentioned, in this paper we assume that the phone
were:
can be rotated only around the Z axis, and consequently, we
Mean assume that there is no change in the acceleration along the Z axis
Standard deviation when performing some activity in the default and non-default
Correlation. position. This means that we do not need to transform the Z
We specified these features, that are calculated using data in time coordinate, just X and Y coordinates. To achieve that, we use a
domain, because we apply the activity recognition system in real- rotation matrix for rotation around the Z axis for an angle θ. The
time locally on a device. For this reason, the features should be rotation matrix is given in (1). To calculate angle θ we use the
relatively simple to compute, to reduce power consumption and information about gravity vectors.
processor load. The selected features do not require signal
representation in the frequency domain, and thus can be computed cos( ) sin( ) 0
relatively fast. Also, the mean is used in the standard deviation R sin( ) cos( ) 0 (1)
calculation, and the mean and the standard deviation are used in 0 0 1
correlation calculation, which further increases computation
speed. By using the gravity vectors from all of the recordings in the
Extracted features were used to train and test 3 classifiers default orientation, we computed an average gravity vector for the
available in the WEKA Data Mining Toolkit [7], which are default orientation (gravity vector is defined as data from the
commonly used in activity recognition [2 – 4, 6]: accelerometer, by three attributes: X, Y and Z). Since the phone’s
C4.5 decision tree screen is facing the user, the Z axis is practically parallel to the
ground, so we take into account only the X and Y components of
Naïve Bayes
the vector.
K-nearest neighbors.
We are mainly interested in the performance of the decision tree
classifier, since it requires the least amount of computation in the
classification phase, which is important when the system is
3
Proceedings IMMoA’13 17 http://www.dbis.rwth-aachen.de/IMMoA2013/
It can be seen that a lot of instances that represent sitting are
classified as standing, which is not intuitive, because these two
activities should be easy to distinguish. This is a consequence of
how WEKA generates the decision tree. When we look at the
generated decision tree shown in Figure 3, we can see that sitting
and standing are distinguished by mean value along the Z axis
(MeanZ). WEKA makes the split on value -9.804189 which is the
maximum value for MeanZ for sitting. It can be expected that the
maximum value for MeanZ will vary for different recordings,
since it can’t be expected from someone to sit in exactly the same
way every time. Since the minimum value for MeanZ for standing
is -1.07289 the split should be done on the value -5.4385, which
is halfway between the maximum for sitting and the minimum for
Figure 1. The accelerometer data collected while walking standing. This would generate a much more robust tree with
higher accuracy. Such decision tree would correctly classify all of
In the classifying phase, we firstly calculate the difference the sitting instances, previously misclassified as standing.
between the angle of the average gravity vector in the default Consequently, decision tree accuracy increases to 92%.
orientation and the angle of the gravity vector for the current
orientation, and this difference is angle θ we need to transform
accelerometer data into the default coordinate system. In the next
step, each sample from the accelerometer is transformed into the
default coordinate system using the rotation matrix given in (1).
Figure 1 shows data from the accelerometer while walking. The
vertical axis represents the acceleration in m/s2. Samples from the
acceleration sensor are represented along the horizontal axis.
Samples 1-250 represent data when the phone is in the default
orientation, samples 251-500 represent data when the phone is in
a non-default orientation and samples 501-750 represent the same Figure 3. Generated decision tree
data as samples 251-500, but transformed into the default
To demonstrate the benefits of this approach, the same classifier
coordinate system. Transformed data is then used to extract
was tested with the same test data, but this time no data
features in the same way as in the mixed orientation data
transformation was performed prior to classification. Based on the
approach. For testing we used data from non-default orientations,
results shown in Table 3, it can be concluded that a classifier built
and data from default orientation. Data from default orientation
using data from only one orientation, cannot classify instances
was treated the same way as data from non-default orientations,
from other orientations with a high success rate. With the
and was transformed accordingly to its gravity vector. We tested
reorientation preprocessing included, the classification accuracy
the same 3 classifiers as in the mixed orientation data approach,
results (shown in Table 2) increase significantly, which
and the results are shown in Table 2.
demonstrates the advantage of this approach, compared to the one
Table 2. Classifier accuracy – Data reorientation assuming a fixed orientation at all times.
preprocessing approach
Table 3. Classifier accuracy – Single orientation classifiers
Classifier Accuracy (in %) without reorientation
C4.5 Decision Tree 86.5 Classifier Accuracy (in %)
Naïve Bayes 91.5 C4.5 Decision Tree 54.4
K-nearest neighbors 95.0 Naïve Bayes 48.5
K-nearest neighbors 58.8
The results obtained are lower than in the mixed orientation data
approach, but are still above 90% threshold, except for the
decision tree classifier. For this reason we analyze the decision To compare the data reorientation preprocessing approach against
tree classifier further. The confusion matrix for the decision tree the mixed orientation data approach, the classifier built with the
classifier is shown in Figure 2. mixed orientation data approach was retested with a data set
consisting of classifier training data, as well as data including
orientations that were not used in classifier training. Accuracy of
the mixed orientation data approach, when handling data
including unknown orientations, can be analyzed in this manner.
Evaluation results are shown in Table 4 and are comparable to the
results achieved with the data reorientation preprocessing
approach. It can be concluded that the mixed orientation data
approach can also handle data from previously unknown
orientations, but with a decrease in accuracy compared to the data
Figure 2. Confusion matrix for the decision tree classifier reorientation preprocessing approach.
4
Proceedings IMMoA’13 18 http://www.dbis.rwth-aachen.de/IMMoA2013/
Table 4. Classifier accuracy - Mixed orientation data approach notice three color groups, where each one, looking from the left to
with unknown orientations the right, represents calculation of mean, standard deviation and
correlation. The box marked with the number 2 represents data
Classifier Accuracy (in %) transformation. We can see how data transformation is performed
C4.5 Decision Tree 85.0 whenever a new sample is read from the accelerometer and that
Naïve Bayes 83.8 feature extraction is performed only when the window shifts for
K-nearest neighbors 88.5 the specified number of samples.
The Figure 5 shows the timeline again but this time in more
6. EVALUATION OF MOBILE CPU LOAD details. Similar to Figure 4, the box marked with the number 1
Using results from the previous test we built an application for
represents mean calculation (green color on the timeline), and the
activity recognition in real-time locally on the device. The
box marked with the number 2 represents data transformation. It
application implements the data reorientation preprocessing
can be seen that data transformation requires a small portion of
approach and uses a prepared decision tree as a classifier. To test
processor time, compared to processor time required to calculate
how much the application and data transformation phase
just one feature, but is performed more often.
participate in the processor load we used DDMS (Dalvik Debug
Monitor Service). The entry point of the application is the onSensorChanged
function, which is called whenever a new sample from the
We ran DDMS for thirty seconds while the application was active
accelerometer is read. This function encapsulates all of the
on the device. The results are shown in Figure 4. The first line of
application processing and participates in the processor load with
the figure represents the main thread of the application in which
22.9%. Also, the function transformData, which encapsulates all
all of the application processing is done. Different methods are
of the data transformation, participates in the processor load with
represented with different colors on the timeline. This figure
1.3%. It can be concluded that although data transformation is
focuses on the period between two feature extractions. The boxes
performed more often than feature extraction, it doesn't increase
marked with the number 1 represent feature calculation. We
processor load significantly.
Figure 4. Result of the DDMS tool applied on the activity recognition application
Figure 5. Result of the DDMS tool in more detail
5
Proceedings IMMoA’13 19 http://www.dbis.rwth-aachen.de/IMMoA2013/
Figure 6. Participation of individual functions in processor load
7. CONCLUSION AND FUTURE WORK 8. ACKNOWLEDGMENTS
In this paper a method for orientation independent activity This paper was realized as a part of the project "Studying climate
recognition from accelerometer data is described. An change and its influence on the environment: impacts, adaptation
accelerometer embedded in a mobile phone was used. Orientation and mitigation" (43007) financed by the Ministry of Education,
of the phone was partially fixed in such a way, that the phone Science and Technological Development of the Republic of
could be rotated only around one axis, but the angle of orientation Serbia within the framework of integrated and interdisciplinary
was arbitrary. To determine the angle of rotation a calibration research for the period 2011-2014.
phase was introduced, in which the user has to stand still for a
couple of seconds with the phone placed in the desired
orientation. In this period the gravity vector is extracted and the
9. REFERENCES
[1] J. Lester, T. Choudhury, G. Booriello. A Practical Approach
difference between the angle of that gravity vector and the angle
to Recognizing Physical Activities. Proceedings of the 4th
of the average gravity vector in the default orientation is
International Conference on Pervasive Computing, 2006, pp.
calculated. This difference is the angle of rotation of the phone.
1-16.
After that the user can start to perform activities freely. The data
from the accelerometer is transformed into the default coordinate [2] L. Bao, S. S. Intille. Activity recognition from user-annotated
system, the features are extracted and the activity recognition is acceleration data. Proceceedings of the 2nd International
performed. Conference on Pervasive Computing, 2004, pp. 1-17.
This method showed slightly reduced accuracy compared to the [3] N. Ravi, N. Dandekar, P. Mysore, M. L. Littman. Activity
method when a classifier is built from data collected from various recognition from accelerometer data. Proceedings of the
orientations, when data from the predefined orientations only is Seventeenth Conference on Innovative Applications of
considered, but the results are still above the threshold (accuracy Artificial Intelligence (IAAI-05), 2005, pp. 1541-1546.
above 90%). When data from not predefined orientations is
considered as well, the proposed method demonstrates increased [4] J. R. Kwapisz, G. M. Weiss, S. A. Moore. Activity
accuracy. The most significant advantage of this method is that it Recognition using Cell Phone Accelerometers. ACM
requires data collection from only one orientation, so less data is SIGKDD Explorations Newsletter, 2010, vol. 12, issue 2, pp.
required for training. Also it makes no assumption on the 74-82.
orientation in the classifying phase; there are no predefined [5] L. Sun, D. Zhang, B. Li, B. Guo, S. Li. Activity Recognition
orientations, so the system will work with data from any on an Accelerometer Embedded Mobile Phone with Varying
orientation. The drawback is the existence of the calibration Positions and Orientations. Proceedings of the 7th
phase, so the process in not fully transparent. This probably International Conference on Ubiquitous Intelligence and
makes this method unusable in some areas of application, like in Computing, 2010, pp. 548-562.
elderly care for example, but in others, like in fitness monitoring,
we believe that is acceptable because the calibration phase is very [6] S. Thiemjarus. A Device-Orientation Independent Method
short and requires very little effort from the user. In return the for Activity Recognition. International Conference on Body
user can place the phone in any orientation. Sensor Networks, 2010, pp. 19 - 23.
In this paper the phone orientation is assumed to be partially [7] I. H. Witte, E. Frank. Data Mining: Practical Machine
fixed, which still limits the user to a certain degree. Demonstrated Learning Tools and Techniques, 3rd ed. Morgan Kaufmann,
method represents only an intermediate step in development of a 2011
method which would acquire a full three-dimensional orientation
in the calibration phase and impose no limits in the phone’s
orientation.
6
Proceedings IMMoA’13 20 http://www.dbis.rwth-aachen.de/IMMoA2013/