A large proportion of the population has become used to sharing private information on the internet with their friends. This information can leak throughout their social network and the extent that personal information propagates depends on the privacy policy of large corporations. In an era of artificial intelligence, data mining, and cloud computing, is it necessary to share personal information with unidentifiable people? Our research shows that deep learning is possible using relatively low capacity computing. The research demonstrates promising results in recognition of human geospatial activity, in prediction of movement, and assessment of contextual risk when applied to spatio-temporal positioning of human subjects. A private surveillance system is thought particularly suitable in the care of those who may, to some, be considered vulnerable.
Advancements in mobile devices that can be worn and carried, their interconnectivity, and the improvement of artificially intelligent tools provide a significant opportunity to assist in the care of the aged. In line with a human right to private life, methods have been examined to keep information private unless there is a moral argument, such as risk to that person, that justifies a breach in privacy. In this scenario, safety is paramount and in the interests of beneficence and non-maleficence an ethical policy in terms of design is employed which defines that personal information is precious; it should therefore not be shared on the internet.
Dementia is a debilitating condition that is growing with the aging society.
Continuance with life in the community is encouraged, since social interaction and physical
activity stimulates a healthy mental state in the person living with symptoms (PwS) and
family carer. (The dyad: Persons living with Dementia (PlwD)). We seek bespoke
Artificially Intelligent solutions for PlwD who wish to preserve independence of the PwS.
Initial system infrastructure and findings are published in [
To this end, a monitoring system is designed that requires the PwS to carry a mobile phone and wear a fitness tracker. It is understood that some may not be comfortable with this. While it is anticipated that the mobile technology component will ultimately be integrated in a single wearable device, these requirements restrict usefulness to PwS who are already comfortable with or habitually carry such devices. 2
The objective is to provide a cost-effective ‘electronic safety net’ that provides peace
of mind to the carer while preserving independence of the PwS. Delayed residential
care can reduce impact economically [
Dementia is caused by a several diseases of the brain. There is a wide spectrum of
symptoms, some of which may manifest in a propensity to walk independently at
inappropriate times [
On-line data privacy divides opinion: many elect to share very varied information about their lives publicly on the internet, but this is not always a conscious decision – ‘small print’ tends to get lost to many users as they install yet another application on their phone. Consent in this way is referred to as informed – the potential for data propagation may be mentioned in supplied information, but few consider this thoroughly.
Leaks of private information have recently been in the headlines; data secured on the cloud is assumed to be safe, but human intervention and inadequate security measures both allow breaches. Advocates of privacy treat their own information very differently and do not share their information with people or organisations – certainly not a computer system. In the case of care for persons who may be considered vulnerable it seems ethically correct that a data protectionist policy should be the default.
The World Health Organisation advocates a human rights-based approach for PlwD [10]. In our study almost two years of data was collected, including GPS, nearby WiFi nodes, activity recognition, even indoor movement and logs of heart rate, steps, and sleep patterns. This monitoring undoubtedly invades the right to private life; the tracking was described by the subject as a big-brother bad dream; on reflection the level of “invasion” depends on who can see the data. 3.1
To reduce costs and to improve potential accessibility in the long term, the equipment used in a working prototype is a standard Android smart-phone and a home-based hub which uses a Quad-Core 1.2Ghz CPU with only 1Gb RAM. Networking between the two in ‘monitoring’ mode is via onboard Bluetooth and Wi-Fi only while at home. 3.2
Deep learning (DL) discovers intricate structure in large datasets, multiple processing layers learn representations of data [11]. Sequential and parallel information is processed in a cyclical (recurrent) fashion by modifying internal weightings of input signals to produce an expected output signal [12, 13]. The hardware platform described may seem restrictive for a DL task in an age where we have got used to resources being server based and ubiquity being the norm. Convention says that DL requires large computing capacity that is not available for the present use case. The goal is that human mobility patterns are learnt, and that perceived risk is assessed against a normality that is ‘safe’. A measure of risk is then used to determine the level of protection required on personal data. Location data for one user is relatively small compared to conventional DL problems; to protect privacy, propagation of this information is restricted to a secure home network. No interaction with the wearable or phone is required of the PwS. 3.3
Long Short-Term Memory (LSTM) networks [14] are a type of RNN suitable for learning and predicting sequential patterns and trends in timelines. Using the on-board accelerometers, they are deployed in human activity recognition (HAR). X-Y-Z accelerometer readings are interpreted over a defined time-period and then compared to those taken in a laboratory to determine probability that a categorised activity is taking place [15]. It has been possible for us to assimilate this using GPS sensor data; a dataset suitable for learning using an LSTM neural network was developed, and the resultant tensor was deployed to the Android platform to provide probability of being on a learnt trajectory or otherwise.
The concept that surveillance need not be invasive is introduced. There is a host of literature relating to HAR [16], there are indoor monitoring studies with AI, e.g. [17], and study of wandering trajectories, e.g. [18]. None describe categorisation of where a person normally goes followed by discrete monitoring that keeps information private until anomalies are found.
Data: GPS data is collected from one subject using a standard HTC-10 smart-phone used just for that purpose. Considerable data preparation is required using the minuteby-minute location co-ordinates. The raw data of one trajectory can be visualised in a histogram as seen in Fig. 1, peaks signifying stops en-route. Total data is compartmentalised based on total movement to date (tm). This is then divided by increment (i) after 2α, for example 1% of tm, is added in both dimensions to allow for noise on the extremities of tm. In Fig. 2. i = 20.
Destinations or latent time at a point can be recognised using centroids of clusters for each trajectory. Once destinations are recognised, it is preferred – for reasons of computational time later – that daily data is reduced for example from 1440 points to only the proportion that represents movement.
Categorisation. Points within each segment (or compartment) are assessed for each trajectory and each segment’s points are compared using a kd-tree based nearest-neighbour algorithm [19]. The degree of similarity is assessed giving a percentage, a threshold gives a similarity decision. There is difficulty in some trajectories where, for example, topographical, atmospheric or networking issues used in test data collection leads to sparse and noisy data. Sparse data was dealt with using 1d-univariate interpolation [20]; this is particularly important in the early days of training where there are few trajectories to compare. Noisy data is essentially ignored at this time by adding a tolerance to the similarity decision just described which is explained in more detail below (Fig. 7.). The result of the comparison algorithm is a segment chain (string) for trajectories with 1 or 0 signifying a match in each square.
Categorisation by comparison of trajectory segment chains by only comparing matched segments significantly reduces the computational capacity required in terms of processing and memory. If a match is found, interpolated point data is added to a master repository with which future comparisons are made. An encoded polyline [21] reduces database size requirements and gives an advantage that trajectories may be stored as an entity. In time the necessity for interpolation is reduced as the repository trajectory density increases.
As seen in Fig. 4. interpolation may cause significant deviation from the route that is travelled, cutting corners and roundabouts, but this level of granulation is considered satisfactory at this point – matching segments rely on a nearest neighbour tolerance (nnT) and merging with subsequent trajectories eventually creates a dense category master.
Fig. 5. a) Increased tolerance leads to a better match
b) Interpolated points are merged to create a dense category master nnT set at 0.005, in decimal degrees which equates to just over 500m is used in the experiments. This tolerance can be linked to tm in further work as the extent of movement defines the granularity required within the movement space. The resultant categories develop into a densely populated polyline seen in Fig. 5b. All movement within a data collection period are matched with destinations recognised in the initial cluster analysis.
In addition to our own, the comparison algorithm was tested using seven users’ data from the Geolife (GL) data-set [22]. This contains better quality GPS trajectories and includes higher variance in modes of travel. With nnT applied to nearest neighbour algorithm it is observed that small deviations from a route are not a significant problem. As can be seen in Fig. 6., four separate tracks converge on a destination and in the extent of this day’s movement all points are within one segment.
Noise, detours and differing distances included in two tracks taking Route 1 and Route 2 in Fig. 7., both arrive at the same place E1 and C2. nnT allows for the eventuality of C1 and D1 not matching Route 2. Adding both to the master increases the possibility that subsequent trajectories match by widening the data-set. Bearing. Some GL users’ data highlighted the difficulty of recognising direction of travel in that only one-way trajectories are recorded. Experimentation with inclusion of direction of travel gave complex results, consequently movement is treated as omnidirectional; the category master is essentially an amalgamation of history on that route. Time factor. This is an important consideration in the study scenario, but the likelihood of a person travelling a recognised trajectory at exactly the same time is low so prediction of this is not required. There are detours from a route, the method of travel may change, there may be traffic. These factors all have a significant impact on spatio-temporal data. Following extensive experimentation, it is concluded that data-point true timestamps cause confusion. Instead, each category master is indexed sequentially. Predictability. Major studies in human mobility patterns find that there is a high degree of temporal and spatial regularity [23]. In the data-sets investigated, this study concurs. Three regularly visited destinations in our data are selected for demonstration; these are travel to University (south), to visit family (west) and to a supermarket (east) seen in Fig. 8. Pre-processing. Category masters are exported and the number of records per category is equalised by interpolating (increased or reduced) to 10,000 records each. Noise is amplified where outliers are interpolated. These outliers will be removed in later versions of the system. The data is stacked and normalised. Train:test split is 80:20. Machine Learning. Inspiration for this is credited to work using Convolutional Neural Network and LSTM RNN in mobile phone HAR applications. The solution selected for our application is Tensorflow ‘BasicLSTMCell’ stacked with ‘MultiRNNCell’ with 64 hidden units. The neural network is expected to learn geo-spatial data to predict categorisation (of the trajectory) when it is given further blocks. Experimentation found that the number of time steps set at 10, in blocks of 10 gave acceptable results over 500 epochs in less than 1.5 hours. Training was carried out using matched trajectories in the GL dataset with similarly acceptable results. Deployment. Using our data-set the resultant tensor is imported to an Android application that sequentially passes arrays of 10 steps of a new trajectory in a timed fashion to a Tensorflow classifier. The probability of the array being Category 1, 2 or 3 gives results for the three trained classes. These predictions are logged on the phone.
Category 1: Correctly predicted with 98-99% certainty unless trajectories overlap. Category 2: Correctly predicted with 55-86% certainty.
Category 3: Correctly predicted with 77- 90% certainty.
a) b) c) Fig. 11. Android category prediction results: The vertical scale on these graphs range from 0 to 1 where 1=100% certainty.
a) Category 1 (dotted line) b) Category 2 (dashed line) c) Category 3 (solid line) The tensor gives reliable prediction of a route being tested in all cases. These are very satisfactory results. Overlap between two categories returning a 50:50 result in the Category 1 test is perfectly acceptable as the routes do overlap. 3.4
In this study the definition of an abnormal location is an important requirement, so an additional category is added specifically for this. A convex hull polygon is formed for each segment in the geo-spatial area of movement, this is built as an output from the classification routine on the hub. If the current location is within a polygon the Tensorflow Classifier result is used in prediction of where the subject is going. If not, then risk is accumulated as follows:
If the subject moves to a new space, the contextual risk of that action is assessed using time and known weather conditions. If this is computed as simply walking in a new area on a sunny afternoon for a short time, this may be appraised as low risk and no information should be shared. In Fig.12 a) movement along the test trajectory is outside known areas (shaded grey), distance and known temperature for the area is monitored (left graph above map). Risk is visualised (right graph): as distance from home (start point) reduces, the system perceives this as returning, and risk is reduced. In Fig.12 b) a detour outside a known path instigates appraisal and logs this as a new place. Risk is reset when probability of normal movement is sensed as seen in Fig. 12 c). In a real-life scenario, retraining will take place when the user returns home. If uncharacteristic movement at a low risk level has been sensed this path is added to the category master. However, if a risk threshold has been breached it is likely that this event is not added as this would require intervention from the carer. Fig. 12. a) Graphed representation of accumulating risk reducing with distance. b) Risk increasing when taking a detour from the trained path. c) Correct categorisation of trajectory with 99% accuracy – accumulated
risk is reset.
The predisposition of the PlwD is assessed using a fitness tracker that monitors sleep disturbances and heart rate. Changes in trends in this data will be used to adjust the sensitivity of the described system in calculating the real-time contextual risk.
The ethical debate regarding the point at which location data is shared and with whom is an interesting area to which our findings may contribute. If time, place or weather is appraised as high risk or ‘inappropriate’; a prior moral framework that rates safety and risk vs. privacy can justify that recent movement and current location may be shared. This sharing can take the form of an SMS alert, or an alert via the internet with a map to a trusted carer. In all other cases the PwS may continue independently. Several questions arise: • When applied to vulnerable persons who may decide this point and who defines what is ‘inappropriate’? • Is normality really ‘safe’? • In production would an AI-based algorithm be trusted? Until now, surveillance of those who may be considered vulnerable lacks legislative control. There are barriers to the use of assistive technology and a DIY approach is being adopted by users [24]. Technological solutions fail to offer a considered technological approach to resolve well-known privacy issues. A private monitoring system that uses AI to determine out-of-the-ordinary movement is novel and since it respects privacy, this surveillance is not intrusive. Development and implementation of such a system is likely to provide PlwD with an electronic safety net that may be used to improve QoL by increasing independent living of the PwS, by providing peace of mind to the carer, and in the ideal case may be used to delay moving to a care home. 5
The research presented in this paper shows promising results both in recognition of human geo-spatial activity and in prediction of movement along normally travelled routes. A cost effective working prototype has been produced to demonstrate that deeplearning techniques can be applied to spatio-temporal data after programmatically categorising normally travelled trajectories. It has been found that when only part of a trajectory has been travelled, likely destinations may be inferred. The application is designed to restrict personal information propagation to a home network. The limitations on computing capacity do not detract from the quality of results.
The World Health Organisation recognises that surveillance is intrusive, that the human rights of PwS are denied and that abuse is present. Locking doors to stop a person eloping violates their human right to liberty, but surveillance is contrary to their human right to private life. Risk, when deviations from known places are sensed, is assessed automatically on a smart-phone in the context of time and weather conditions. Human rights (of private life and liberty) of the person with symptoms will be respected until the point at which it is judged that a prior moral argument of safety and risk supersedes the importance of privacy. If this happens, alerts containing location and recent movements are shared with an assigned carer, thus facilitating swift recovery.
The potential of the AI system described here is considerable; it is likely that many that value the importance of privacy highly will welcome a surveillance system that monitors but does not divulge detail. Predictions of likely trajectory of movement using real-time location data is novel, as is the concept of private surveillance as described. Availability of an internet connection or at least cellular coverage to deliver alerts is a requirement for an implementation of this concept.
Ongoing work includes the processing of GPS data from recruited volunteers. The assessment of complex and intertwined trajectories and comparison of different scales of movement for one user will be carried out. A bespoke application will be deployed for testing in the field by the volunteers. Findings will contribute to further dissemination after consultation with health professionals and PlwD.
Knowledge Economy Skills Scholarships (KESS) is a pan-Wales higher level skills initiative led by Bangor University on behalf of the HE sectors in Wales. It is part funded by the Welsh Government's European Social Fund (ESF) convergence programme for West Wales and the Valleys and is supported by an industrial partner: SymlConnect Limited. [9] Riikonen, M., Mäkelä, K. and Perälä, S. (2010). Safety and monitoring technologies for the homes of people with dementia. Gerontechnology, 9(1), pp. 32-45.
doi:http://dx.doi.org/10.4017/gt.2010.09.01.003.00 [10] World Health Organisation. (2015). Ensuring a human rights-based approach for people living with dementia. Retrieved Mar 2018, from The need for a human-rights based approach: http:// www.who.int/mental_health/neurology/dementia/dementia_thematicbrief_human_rights.pdf [11] LeCun, Y., Bengio, Y. and Hinton, G. (2015). Deep learning. Nature (521), 436–444.
doi:10.1038/nature14539 [12] Schmidhuber, J. (2014). Deep Learning in Neural Networks: An Overview. CoRR
Neural Networks, pp 85-117. doi:10.1016/j.neunet.2014.09.003 [13] Brownlee, J. (2016). How to Implement the Backpropagation Algorithm From Scratch In Python. Retrieved Mar 2018, from Machine Learning Mastery: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ [14] Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735-1780. doi:10.1162/neco.1997.9.8.1735 [15] Kwapisz, J. R., Weiss, G. M., Moore, S. A. (2010). Activity Recognition using Cell Phone Accelerometers. ACM SIGKDD Explorations Newsletter, 12(2), 74-82.
doi:10.1145/1964897.1964918 [16] Ramamurthy, S. R., Roy, N. (2018). Recent trends in machine learning for human activity recognition—A survey. Data mining and Knowledge Discovery, 1-19. doi:10.1002/widm.1254 [17] Arifoglu, D., and Bouchachia, A. (2017). Activity Recognition and Abnormal Behaviour Detection with Recurrent Neural Networks. 14th International Conference on Mobile Systems and Pervasive Computing (MobiSPC 2017). 110, pp. 86-93. Procedia Computer Science. doi: 10.1016/j.procs.2017.06.121 [18] E. Batista, F. Borras, F. Casino and A. Solanas. (2015). A study on the detection of wandering patterns in human trajectories. 6th International Conference on Information, Intelligence, Systems and Applications (IISA) (pp. 1-6). Corfu, Greece: IEEE. doi:10.1109/IISA.2015.7387995 [19] Maneewongvatana, S. and Mount, D. M. (1999). Analysis of approximate nearest neighbor searching with clustered point sets. CoRR. doi:arXiv:cs/9901013v1 [20] SciPy.Org. (2017, Oct 25). Interpolation (scipy.interpolate). Retrieved March 17, 2018, from https://docs.scipy.org/doc/scipy/reference/interpolate.html [21] Google. (2018, April 13). Encoded Polyline Algorithm Format. Retrieved Mar 17, 2018, from Google https://developers.google.com/maps/documentation/utilities/polylinealgorithm?csw=1 [22] Zheng et al. (2011). Geolife GPS trajectories 1.1. In Geolife GPS trajectory dataset - User Guide.
Retrieved from https://www.microsoft.com/en-us/download/details.aspx?id=52367 [23] González, M. C., Hidalgo, C. A. and Barabási, A. (2008, June). Understanding individual human mobility patterns. Nature (453), pp. 779–782. doi:doi:10.1038/nature06958 [24] Gibson et al. (2015). The everyday use of assistive technology by people with dementia and their family carers: a qualitative study. BMC Geriatrics, 15, 89. doi:10.1186/s12877-015-0091-3