=Paper=
{{Paper
|id=Vol-2142/paper14
|storemode=property
|title=Preserving safety, privacy and mobility of persons living with Dementia by recognising uncharacteristic out-door movement using Recurrent Neural Networks with low computing capacity
|pdfUrl=https://ceur-ws.org/Vol-2142/paper14.pdf
|volume=Vol-2142
|authors=Steve Williams,J Mark Ware,Berndt Müller
|dblpUrl=https://dblp.org/rec/conf/ijcai/WilliamsWM18
}}
==Preserving safety, privacy and mobility of persons living with Dementia by recognising uncharacteristic out-door movement using Recurrent Neural Networks with low computing capacity==
https://ceur-ws.org/Vol-2142/paper14.pdf
Preserving safety, privacy and mobility of persons living
with Dementia by recognising uncharacteristic out-door
movement using Recurrent Neural Networks with low
computing capacity
Steve Williams , J Mark Ware , Berndt Müller
1 1 1
1
Faculty of Computing, Engineering and Science, University of South Wales, UK
{steve.williams1,mark.ware,bertie.muller}@southwales.ac.uk
A large proportion of the population has become used to sharing private infor-
mation on the internet with their friends. This information can leak throughout
their social network and the extent that personal information propagates depends
on the privacy policy of large corporations. In an era of artificial intelligence,
data mining, and cloud computing, is it necessary to share personal information
with unidentifiable people? Our research shows that deep learning is possible
using relatively low capacity computing. The research demonstrates promising
results in recognition of human geospatial activity, in prediction of movement,
and assessment of contextual risk when applied to spatio-temporal positioning of
human subjects. A private surveillance system is thought particularly suitable in
the care of those who may, to some, be considered vulnerable.
Keywords: privacy, deep learning, assisted-living, mobile computing, ethics,
mHeath, wearable health, dementia, safer walking, GPS, LSTM, RNN
1 Background
Advancements in mobile devices that can be worn and carried, their interconnectiv-
ity, and the improvement of artificially intelligent tools provide a significant oppor-
tunity to assist in the care of the aged. In line with a human right to private life, meth-
ods have been examined to keep information private unless there is a moral argument,
such as risk to that person, that justifies a breach in privacy. In this scenario, safety is
paramount and in the interests of beneficence and non-maleficence an ethical policy in
terms of design is employed which defines that personal information is precious; it
should therefore not be shared on the internet.
Dementia is a debilitating condition that is growing with the aging society. Contin-
uance with life in the community is encouraged, since social interaction and physical
activity stimulates a healthy mental state in the person living with symptoms (PwS) and
family carer. (The dyad: Persons living with Dementia (PlwD)). We seek bespoke Ar-
tificially Intelligent solutions for PlwD who wish to preserve independence of the PwS.
Initial system infrastructure and findings are published in [1], the suitability of a mobile
computer technology in tracking PwS and ethical aspects are previously discussed in
[2]. The work described here contributes to the ethical debate regarding at which point
information gathered when monitoring a PwS should be shared by introducing a tech-
nological solution that keeps data private until a threshold of risk to health is reached.
To this end, a monitoring system is designed that requires the PwS to carry a mobile
phone and wear a fitness tracker. It is understood that some may not be comfortable
with this. While it is anticipated that the mobile technology component will ultimately
be integrated in a single wearable device, these requirements restrict usefulness to PwS
who are already comfortable with or habitually carry such devices.
2 The problem
The objective is to provide a cost-effective ‘electronic safety net’ that provides peace
of mind to the carer while preserving independence of the PwS. Delayed residential
care can reduce impact economically [3], but more importantly this can sustain the fun-
damentals of a family unit.
2.1 Dementia
Dementia is caused by a several diseases of the brain. There is a wide spectrum of
symptoms, some of which may manifest in a propensity to walk independently at inap-
propriate times [4]. Literature indicates that this can lead to premature mortality [4, 5,
6]. Actions to mitigate this risk can lead to increased dependence, to curtailment of
social activities, and reduction in quality of life (QoL) [7]. Elopement episodes are a
major reason for nursing home admission [8]; one study in Finland reports that this may
be delayed, using assistive technology, by an average of eight months [9].
2.2 Privacy
On-line data privacy divides opinion: many elect to share very varied information
about their lives publicly on the internet, but this is not always a conscious decision –
‘small print’ tends to get lost to many users as they install yet another application on
their phone. Consent in this way is referred to as informed – the potential for data prop-
agation may be mentioned in supplied information, but few consider this thoroughly.
Leaks of private information have recently been in the headlines; data secured on the
cloud is assumed to be safe, but human intervention and inadequate security measures
both allow breaches. Advocates of privacy treat their own information very differently
and do not share their information with people or organisations – certainly not a com-
puter system. In the case of care for persons who may be considered vulnerable it seems
ethically correct that a data protectionist policy should be the default.
3 A human rights-based approach
The World Health Organisation advocates a human rights-based approach for PlwD
[10]. In our study almost two years of data was collected, including GPS, nearby Wi-
Fi nodes, activity recognition, even indoor movement and logs of heart rate, steps, and
sleep patterns. This monitoring undoubtedly invades the right to private life; the track-
ing was described by the subject as a big-brother bad dream; on reflection the level of
“invasion” depends on who can see the data.
3.1 Cost effective hardware platform
To reduce costs and to improve potential accessibility in the long term, the equipment
used in a working prototype is a standard Android smart-phone and a home-based hub
which uses a Quad-Core 1.2Ghz CPU with only 1Gb RAM. Networking between the
two in ‘monitoring’ mode is via onboard Bluetooth and Wi-Fi only while at home.
3.2 Unconventional Deep Learning
Deep learning (DL) discovers intricate structure in large datasets, multiple processing
layers learn representations of data [11]. Sequential and parallel information is pro-
cessed in a cyclical (recurrent) fashion by modifying internal weightings of input sig-
nals to produce an expected output signal [12, 13]. The hardware platform described
may seem restrictive for a DL task in an age where we have got used to resources being
server based and ubiquity being the norm. Convention says that DL requires large
computing capacity that is not available for the present use case. The goal is that human
mobility patterns are learnt, and that perceived risk is assessed against a normality that
is ‘safe’. A measure of risk is then used to determine the level of protection required on
personal data. Location data for one user is relatively small compared to conventional
DL problems; to protect privacy, propagation of this information is restricted to a secure
home network. No interaction with the wearable or phone is required of the PwS.
3.3 Deep Learning using Recurrent Neural Networks (RNN)
Long Short-Term Memory (LSTM) networks [14] are a type of RNN suitable for
learning and predicting sequential patterns and trends in timelines. Using the on-board
accelerometers, they are deployed in human activity recognition (HAR). X-Y-Z accel-
erometer readings are interpreted over a defined time-period and then compared to
those taken in a laboratory to determine probability that a categorised activity is taking
place [15]. It has been possible for us to assimilate this using GPS sensor data; a dataset
suitable for learning using an LSTM neural network was developed, and the resultant
tensor was deployed to the Android platform to provide probability of being on a learnt
trajectory or otherwise.
The concept that surveillance need not be invasive is introduced. There is a host of
literature relating to HAR [16], there are indoor monitoring studies with AI, e.g. [17],
and study of wandering trajectories, e.g. [18]. None describe categorisation of where a
person normally goes followed by discrete monitoring that keeps information private
until anomalies are found.
Data: GPS data is collected from one subject using a standard HTC-10 smart-phone
used just for that purpose. Considerable data preparation is required using the minute-
by-minute location co-ordinates. The raw data of one trajectory can be visualised in a
histogram as seen in Fig. 1, peaks signifying stops en-route. Total data is compart-
mentalised based on total movement to date (tm). This is then divided by increment (i)
after 2α, for example 1% of tm, is added in both dimensions to allow for noise on the
extremities of tm. In Fig. 2. i = 20.
Fig. 1. Spatio-temporal raw data, time at Fig. 2. Boundaries of the extent of total move-
a place is represented as peaks with ment for 3 months, i = 20.
movement represented as single points. Map data: © Google.
Destinations or latent time at a point can be recognised using centroids of clusters
for each trajectory. Once destinations are recognised, it is preferred – for reasons of
computational time later – that daily data is reduced for example from 1440 points to
only the proportion that represents movement.
Categorisation. Points within each segment (or compartment) are assessed for each
trajectory and each segment’s points are compared using a kd-tree based nearest-neigh-
bour algorithm [19]. The degree of similarity is assessed giving a percentage, a thresh-
old gives a similarity decision. There is difficulty in some trajectories where, for exam-
ple, topographical, atmospheric or networking issues used in test data collection leads
to sparse and noisy data. Sparse data was dealt with using 1d-univariate interpolation
[20]; this is particularly important in the early days of training where there are few
trajectories to compare. Noisy data is essentially ignored at this time by adding a tol-
erance to the similarity decision just described which is explained in more detail below
(Fig. 7.). The result of the comparison algorithm is a segment chain (string) for trajec-
tories with 1 or 0 signifying a match in each square.
Fig. 3. A successful match of two segment chains. Fig. 4. Interpolation used to deal with
29 segments, i=10 sparse data causing accuracy issues
Map data: © Google
Categorisation by comparison of trajectory segment chains by only comparing
matched segments significantly reduces the computational capacity required in terms
of processing and memory. If a match is found, interpolated point data is added to a
master repository with which future comparisons are made. An encoded polyline [21]
reduces database size requirements and gives an advantage that trajectories may be
stored as an entity. In time the necessity for interpolation is reduced as the repository
trajectory density increases.
As seen in Fig. 4. interpolation may cause significant deviation from the route that
is travelled, cutting corners and roundabouts, but this level of granulation is considered
satisfactory at this point – matching segments rely on a nearest neighbour tolerance
(nnT) and merging with subsequent trajectories eventually creates a dense category
master.
Fig. 5. a) Increased tolerance leads to a better match
b) Interpolated points are merged to create a dense category master
nnT set at 0.005, in decimal degrees which equates to just over 500m is used in the
experiments. This tolerance can be linked to tm in further work as the extent of move-
ment defines the granularity required within the movement space. The resultant cate-
gories develop into a densely populated polyline seen in Fig. 5b. All movement within
a data collection period are matched with destinations recognised in the initial cluster
analysis.
In addition to our own, the comparison algorithm was tested using seven users’ data
from the Geolife (GL) data-set [22]. This contains better quality GPS trajectories and
includes higher variance in modes of travel. With nnT applied to nearest neighbour
algorithm it is observed that small deviations from a route are not a significant problem.
As can be seen in Fig. 6., four separate tracks converge on a destination and in the
extent of this day’s movement all points are within one segment.
Noise, detours and differing distances included in two tracks taking Route 1 and
Route 2 in Fig. 7., both arrive at the same place E1 and C2. nnT allows for the eventu-
ality of C1 and D1 not matching Route 2. Adding both to the master increases the pos-
sibility that subsequent trajectories match by widening the data-set.
Fig. 6. Walking via different routes to a bus- Fig. 7. Widening the category master by al-
stop Map data: © GoogleMaps lowing a nearest neighbour tolerance
Bearing. Some GL users’ data highlighted the difficulty of recognising direction of
travel in that only one-way trajectories are recorded. Experimentation with inclusion
of direction of travel gave complex results, consequently movement is treated as omni-
directional; the category master is essentially an amalgamation of history on that route.
Time factor. This is an important consideration in the study scenario, but the likelihood
of a person travelling a recognised trajectory at exactly the same time is low so predic-
tion of this is not required. There are detours from a route, the method of travel may
change, there may be traffic. These factors all have a significant impact on spatio-tem-
poral data. Following extensive experimentation, it is concluded that data-point true
timestamps cause confusion. Instead, each category master is indexed sequentially.
Predictability. Major studies in human mobility patterns find that there is a high de-
gree of temporal and spatial regularity [23]. In the data-sets investigated, this study
concurs. Three regularly visited destinations in our data are selected for demonstration;
these are travel to University (south), to visit family (west) and to a supermarket (east)
seen in Fig. 8.
Fig. 8. Three categories overlaid, interpolated; 3 x 10k records
Pre-processing. Category masters are exported and the number of records per category
is equalised by interpolating (increased or reduced) to 10,000 records each. Noise is
amplified where outliers are interpolated. These outliers will be removed in later ver-
sions of the system. The data is stacked and normalised. Train:test split is 80:20.
Machine Learning. Inspiration for this is credited to work using Convolutional Neural
Network and LSTM RNN in mobile phone HAR applications. The solution selected
for our application is Tensorflow ‘BasicLSTMCell’ stacked with ‘MultiRNNCell’ with
64 hidden units. The neural network is expected to learn geo-spatial data to predict
categorisation (of the trajectory) when it is given further blocks. Experimentation
found that the number of time steps set at 10, in blocks of 10 gave acceptable results
over 500 epochs in less than 1.5 hours.
Fig. 9. LSTM training session over 1.4 hours. 90-97% accuracy
Training was carried out using matched trajectories in the GL dataset with similarly
acceptable results.
Fig. 10. LSTM training session for 7 users (in 9 tests) using Geolife data-set
Deployment. Using our data-set the resultant tensor is imported to an Android appli-
cation that sequentially passes arrays of 10 steps of a new trajectory in a timed fashion
to a Tensorflow classifier. The probability of the array being Category 1, 2 or 3 gives
results for the three trained classes. These predictions are logged on the phone.
Mobile Results.
Category 1: Correctly predicted with 98-99% certainty unless trajectories overlap.
Category 2: Correctly predicted with 55-86% certainty.
Category 3: Correctly predicted with 77- 90% certainty.
a) b) c)
Fig. 11. Android category prediction results: The vertical scale on these graphs range from 0
to 1 where 1=100% certainty.
a) Category 1 (dotted line)
b) Category 2 (dashed line)
c) Category 3 (solid line)
The tensor gives reliable prediction of a route being tested in all cases. These are very
satisfactory results. Overlap between two categories returning a 50:50 result in the
Category 1 test is perfectly acceptable as the routes do overlap.
3.4 Inferring an unknown location
In this study the definition of an abnormal location is an important requirement, so an
additional category is added specifically for this. A convex hull polygon is formed for
each segment in the geo-spatial area of movement, this is built as an output from the
classification routine on the hub. If the current location is within a polygon the Tensor-
flow Classifier result is used in prediction of where the subject is going. If not, then risk
is accumulated as follows:
If the subject moves to a new space, the contextual risk of that action is assessed
using time and known weather conditions. If this is computed as simply walking in a
new area on a sunny afternoon for a short time, this may be appraised as low risk and
no information should be shared. In Fig.12 a) movement along the test trajectory is
outside known areas (shaded grey), distance and known temperature for the area is
monitored (left graph above map). Risk is visualised (right graph): as distance from
home (start point) reduces, the system perceives this as returning, and risk is reduced.
In Fig.12 b) a detour outside a known path instigates appraisal and logs this as a new
place. Risk is reset when probability of normal movement is sensed as seen in Fig. 12
c). In a real-life scenario, retraining will take place when the user returns home. If
uncharacteristic movement at a low risk level has been sensed this path is added to the
category master. However, if a risk threshold has been breached it is likely that this
event is not added as this would require intervention from the carer.
a) b) c)
Fig. 12. a) Graphed representation of accumulating risk reducing with distance.
b) Risk increasing when taking a detour from the trained path.
c) Correct categorisation of trajectory with 99% accuracy – accumulated
risk is reset.
The predisposition of the PlwD is assessed using a fitness tracker that monitors sleep
disturbances and heart rate. Changes in trends in this data will be used to adjust the
sensitivity of the described system in calculating the real-time contextual risk.
4 Ethics
The ethical debate regarding the point at which location data is shared and with
whom is an interesting area to which our findings may contribute. If time, place or
weather is appraised as high risk or ‘inappropriate’; a prior moral framework that rates
safety and risk vs. privacy can justify that recent movement and current location may
be shared. This sharing can take the form of an SMS alert, or an alert via the internet
with a map to a trusted carer. In all other cases the PwS may continue independently.
Several questions arise:
• When applied to vulnerable persons who may decide this point and who de-
fines what is ‘inappropriate’?
• Is normality really ‘safe’?
• In production would an AI-based algorithm be trusted?
Until now, surveillance of those who may be considered vulnerable lacks legislative
control. There are barriers to the use of assistive technology and a DIY approach is
being adopted by users [24]. Technological solutions fail to offer a considered techno-
logical approach to resolve well-known privacy issues. A private monitoring system
that uses AI to determine out-of-the-ordinary movement is novel and since it respects
privacy, this surveillance is not intrusive. Development and implementation of such a
system is likely to provide PlwD with an electronic safety net that may be used to im-
prove QoL by increasing independent living of the PwS, by providing peace of mind to
the carer, and in the ideal case may be used to delay moving to a care home.
5 Conclusion
The research presented in this paper shows promising results both in recognition of
human geo-spatial activity and in prediction of movement along normally travelled
routes. A cost effective working prototype has been produced to demonstrate that deep-
learning techniques can be applied to spatio-temporal data after programmatically cat-
egorising normally travelled trajectories. It has been found that when only part of a
trajectory has been travelled, likely destinations may be inferred. The application is
designed to restrict personal information propagation to a home network. The limita-
tions on computing capacity do not detract from the quality of results.
The World Health Organisation recognises that surveillance is intrusive, that the hu-
man rights of PwS are denied and that abuse is present. Locking doors to stop a person
eloping violates their human right to liberty, but surveillance is contrary to their human
right to private life. Risk, when deviations from known places are sensed, is assessed
automatically on a smart-phone in the context of time and weather conditions. Human
rights (of private life and liberty) of the person with symptoms will be respected until
the point at which it is judged that a prior moral argument of safety and risk supersedes
the importance of privacy. If this happens, alerts containing location and recent move-
ments are shared with an assigned carer, thus facilitating swift recovery.
The potential of the AI system described here is considerable; it is likely that many
that value the importance of privacy highly will welcome a surveillance system that
monitors but does not divulge detail. Predictions of likely trajectory of movement using
real-time location data is novel, as is the concept of private surveillance as described.
Availability of an internet connection or at least cellular coverage to deliver alerts is a
requirement for an implementation of this concept.
Ongoing work includes the processing of GPS data from recruited volunteers. The
assessment of complex and intertwined trajectories and comparison of different scales
of movement for one user will be carried out. A bespoke application will be deployed
for testing in the field by the volunteers. Findings will contribute to further dissemina-
tion after consultation with health professionals and PlwD.
Acknowledgements
Knowledge Economy Skills Scholarships (KESS) is a pan-Wales higher level skills initiative led by Bangor
University on behalf of the HE sectors in Wales. It is part funded by the Welsh Government's European
Social Fund (ESF) convergence programme for West Wales and the Valleys and is supported by an industrial
partner: SymlConnect Limited.
References
[1] Williams, S., Müller, B. (2017). Agents and Dementia - Smart Risk Assessment, 14th European
Conference on Multi-Agent Systems, EUMAS 2016 pp. 277-284 ISBN 978-3-319-59294-7).
Valencia, Spain: Springer.
[2] Williams, S., Ware, J. M. (2015). Is the use of 'mobile computer technology' appropriate for locating
people with dementia? GIS Research UK (GISRUK) 2015 Proceedings pp. 659-664. Leeds:
Figshare. doi:10.6084/m9.figshare.1491375
[3] Curtis, L., Burns, A. (2015). Unit Costs of Health and Social Care 2015. Canterbury, UK: Personal
Social Services Research Unit. Retrieved August 22nd, 2016, from Personal Social Service
Research Unit: http://www.pssru.ac.uk/project-pages/unit-costs/2015/index.php
[4] Silverstein, N. F. et al. (2006). Dementia and Wandering Behavior: Concern for the Lost Elder. New
York: Springer Publishing Company ISBN 0-8261-0272-7.
[5] McShane R. et al. (1998). Getting Lost in Dementia: A Longitudinal Study of a Behavioral Symptom.
International Psychogeriatrics Volume 10 / Issue 03, pp. 253-260. Retrieved from
http://www.ncbi.nlm.nih.gov/pubmed/9785146
[6] Ali, N. et al. (2016). Risk assessment of wandering behavior in mild dementia. Int J Geriatr
Psychiatry(31), 367–374. doi:https://doi.org/10.1002/gps.4336
[7] Martyr, A. et al. Living Well with Dementia: A systematic review. Alzheimer's & Dementia: The
Journal of the Alzheimer's Association, 13(7), pp. 1567-1568 . doi:10.1016/j.jalz.2017.07.725
[8] Cipriani, G., Lucetti, C., Nuti, A. and Danti, S. (2014). Wandering and dementia. Psychogeriatrics, 14,
135–142. doi:doi:10.1111/psyg.12044
[9] Riikonen, M., Mäkelä, K. and Perälä, S. (2010). Safety and monitoring technologies for the homes of
people with dementia. Gerontechnology, 9(1), pp. 32-45.
doi:http://dx.doi.org/10.4017/gt.2010.09.01.003.00
[10] World Health Organisation. (2015). Ensuring a human rights-based approach for people living with
dementia. Retrieved Mar 2018, from The need for a human-rights based approach: http://
www.who.int/mental_health/neurology/dementia/dementia_thematicbrief_human_rights.pdf
[11] LeCun, Y., Bengio, Y. and Hinton, G. (2015). Deep learning. Nature (521), 436–444.
doi:10.1038/nature14539
[12] Schmidhuber, J. (2014). Deep Learning in Neural Networks: An Overview. CoRR
Neural Networks, pp 85-117. doi:10.1016/j.neunet.2014.09.003
[13] Brownlee, J. (2016). How to Implement the Backpropagation Algorithm From Scratch
In Python. Retrieved Mar 2018, from Machine Learning Mastery: https://machinelearningmas-
tery.com/implement-backpropagation-algorithm-scratch-python/
[14] Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8),
1735-1780. doi:10.1162/neco.1997.9.8.1735
[15] Kwapisz, J. R., Weiss, G. M., Moore, S. A. (2010). Activity Recognition using Cell Phone
Accelerometers. ACM SIGKDD Explorations Newsletter, 12(2), 74-82.
doi:10.1145/1964897.1964918
[16] Ramamurthy, S. R., Roy, N. (2018). Recent trends in machine learning for human activity
recognition—A survey. Data mining and Knowledge Discovery, 1-19. doi:10.1002/widm.1254
[17] Arifoglu, D., and Bouchachia, A. (2017). Activity Recognition and Abnormal Behaviour Detection
with Recurrent Neural Networks. 14th International Conference on Mobile Systems and
Pervasive Computing (MobiSPC 2017). 110, pp. 86-93. Procedia Computer Science. doi:
10.1016/j.procs.2017.06.121
[18] E. Batista, F. Borras, F. Casino and A. Solanas. (2015). A study on the detection of wandering patterns
in human trajectories. 6th International Conference on Information, Intelligence, Systems and
Applications (IISA) (pp. 1-6). Corfu, Greece: IEEE. doi:10.1109/IISA.2015.7387995
[19] Maneewongvatana, S. and Mount, D. M. (1999). Analysis of approximate nearest neighbor searching
with clustered point sets. CoRR. doi:arXiv:cs/9901013v1
[20] SciPy.Org. (2017, Oct 25). Interpolation (scipy.interpolate). Retrieved March 17, 2018, from
https://docs.scipy.org/doc/scipy/reference/interpolate.html
[21] Google. (2018, April 13). Encoded Polyline Algorithm Format. Retrieved Mar 17, 2018, from Google
https://developers.google.com/maps/documentation/utilities/polylinealgorithm?csw=1
[22] Zheng et al. (2011). Geolife GPS trajectories 1.1. In Geolife GPS trajectory dataset - User Guide.
Retrieved from https://www.microsoft.com/en-us/download/details.aspx?id=52367
[23] González, M. C., Hidalgo, C. A. and Barabási, A. (2008, June). Understanding individual human
mobility patterns. Nature (453), pp. 779–782. doi:doi:10.1038/nature06958
[24] Gibson et al. (2015). The everyday use of assistive technology by people with dementia and their family
carers: a qualitative study. BMC Geriatrics, 15, 89. doi:10.1186/s12877-015-0091-3