Automatically determining anomalies in human behavior is an important tool in ambient assisted living, especially when it concerns elderly people that for several reasons cannot be continuously monitored and assisted by a caregiver or a family member. This paper proposes a network of low cost RGB-D sensors with no overlapping elds-of-view, capable of identifying anomalous behaviors with respect a pre-learned normal one. A 3D trajectory analysis is carried out by comparing three di erent classi ers (SVM, neural networks and k-nearest neighbors). The results on real experiments prove the e ectiveness of the proposed approach both in terms of performances and of real time application. Moreover, the possibility to extract and use depth information without considering color information enables the system to operate while respecting user privacy.
Ageing of population is predicted to be a serious global concern in the near future
and it is already an important challenge in some countries. This is going to have
important implications in many aspects of our life and will dramatically a ect
policies around the world to address the issue, including its impact on health
care expenditure [13]. In this context, extending the time in which elderly people
can act independently, without the need to be housed in assisted living facilities
or be hospitalized, is not only humanly desirable but required for economic
sustainability [24]. Cameras have proven to be an important technology in di erent
surveillance applications, used in a rapidly growing industry. Many factors
contribute to this trend such as escalating safety and security concerns, decreasing
hardware costs and advances in processing and storage capabilities. These
advances have enabled automatic tools for monitoring both small and vast areas.
Initially focused on security applications[
Video surveillance applications employ object detection algorithms, along with higher-level processing, such as tracking or event analysis, to extract meaningful data from the captured scenes. Detection algorithms vary in relation to the task to be performed and the particular context. Most of the time their focus is on moving objects in an otherwise static environment, where a background can be dynamically modelled and updated and moving objects are then obtained through background subtraction techniques. However, these techniques are in uenced by illumination conditions, performing poorly both when images appear overly bright and saturated, and when captured scenes are dimly lit. Arti cial lights can also prove challenging since lightbulbs icker due to alternate current, with consequences on the background modelling, although some solutions can be employed to counter it[19].
Part-based human body detectors based on color information have been
proposed by [22] where SVM classi ers are used to learn speci c parts of the human
body on a variety of poses and backgrounds. This approach is able to handle
partial occlusions enabling robust identi cation and tracking in crowded scenes.
Nie et al. [14] developed an algorithm for tracklets association with overlapping
objects for crowded scenes. Occlusions are handled using a part-based similarity
algorithm while the tracklets association is formulated as a Maximum A
Posterior problem by using a Markov-chain with spatiotemporal context constraints.
In [
The challenges related to the employment of RGB sensors, even when used
with stereo algorithms, have led researchers to investigate other sensors, such as
time-of- ight cameras [
In recent times, novel camera systems such as Microsoft Kinect, pushed by
research advances and economies of scale, have enabled a widespread
development of 3D vision algorithms that can operate on RGB-D data [
In [
In addition to tracking, event analysis is another major requirement in most surveillance applications. It can be approached either with high-level semantic interpretation of video sequences or by performing anomaly detection, by subdividing sequences in normal and abnormal sets and employing classi cation techniques to learn a model able to discriminate between them. In [17] Piciarelli et al. follow this approach by using a single class support vector machine able to identify anomalous trajectory.
The work presented in this paper approaches the event analysis problem by learning a model. In this case walking behaviors are split between normal and anomalous ones by relying on trajectory evaluation focusing on the detection of wandering situations. Wandering is indeed an important hint in the assisted living context since it is one of the symptoms shown by people a ected by dementia[23].
The system, designed for an indoor environment, uses multiple Kinect cameras, suitably placed in a corridor for maximum coverage and no overlapping. Skeletal features are extracted from the RGB-D sensor by exploiting the OpenNI framework and by considering the extracted torso feature. A proper Kalman lter is used for the prediction step and allows robust people tracking both inter-camera and intra-camera. Trajectories are assembled together in a common reference system, by extrapolating the path using splines. Finally, anomaly behavior detection is performed using di erent classi cation algorithms by comparing multiple techniques: in addition to an SVM classi er, we use a k-NN algorithm and a feed forward neural network trained with a backpropagation algorithm. Additional information about the methodology are reported in Section 2, while experimental results follows in Section 3. Conclusions and consideration on future researches are drawn in Section 4. 2
The methodology proposed in this paper can be summarized in three main blocks, namely: 1. 3D Data acquisition and Preprocessing; 2. Feature Extraction; 3. Behavior Classi cation.
Data coming from one or multiple RGB-D sensors is initially acquired and pre-processed to obtain three dimensional trajectories of a moving subject. Then, a speci c set of features is extracted from each trajectory in order to perform the classi cation task and understand if it leads to an anomalous behavior or not. 2.1
In the rst step, several RGB-D sensors with no overlapping elds of view are
employed to acquire depth data from an observed scene. In order to refer the
depth maps produced by each sensor, or equivalently the corresponding point
clouds, into a global reference system it is necessary to perform a preliminary
calibration phase. Several reference points, whose coordinates in a global
reference system are already known, are observed in each camera and are used to
determine the transformation matrices between the local reference systems and
the global one. In particular, by considering the position of every sample points
in the local reference systems CKi , i = 1; 2; ::n, of the n RGB-D sensors, it is
possible to nd the 4 4 matrices Mi which are able to transform every point
pCKi = [xpKi , ypKi , zpKi ; 1]T , de ned in CKi , into the global reference system C,
since pC = [xp; yp; zp; 1]T = Mi pCKi . Solutions are obtained in the least-squares
(LS) sense through the application of a standard registration algorithm, based
on the single value decomposition[
Once all cameras refer to the same system of coordinates, people have to
be detected and then tracked in time. As we will describe in the next session,
in this paper we use the OpenNI framework to detect people and recover their
3D positions in each frame. Since people can move into an extended region,
performing complex movements, which are not completely under the eld of
view of a single camera, it is necessary to put in a unique trajectory the 3D
points generated by a user in each camera. For this reason a Kalman lter[
Following the diagram in Fig. 1 for every frame at a speci c time t, the user detection procedure segments new Q users (Q 0) in the elds of view of the n sensors placed in the environment. On the other hand, Nt 1 users (Nt 1 0) were computed at the previous discrete time instant t 1. The task of user tracking aims to associate, if possible, users detected at time t with those identi ed at time t 1.
We suppose that each user detected at t 1 moves with constant velocity. Its position is thus predicted by using a Kalman lter, which operates over a state vector de ned by the position and the speed of the user. The predicted positions of the Nt 1 users are thus compared with those observed in the environment. This comparison is mediated by a cost computation, easily de ned in terms of the Euclidean distance between the positions of every current user and the Nt 1 previous ones. Users with close positions, i.e. with small cost values, are in relation. Finally, for each reassigned user, the state of the Kalman lter is updated in order to reduce the contribution of measurement noise. This strategy is applied between every pair of consecutive frames, where in general three di erent events can arise: { Users are still visible in the eld of view of the speci c sensor and thus are correctly assigned to corresponding new instances observed in the scene. In this case the Kalman lter operates to correct measurement, in accordance with the previous estimation; { New users enter in the scene and are detected in the current frame t. New instances are then initialized with the states of the detected users; { Users are lost and no longer visible in the elds of view of the sensors. The states of the lost users are still kept in the analysis and evolve following the model of the Kalman lter, i.e. at constant velocity.
As a result of this processing, each user is tracked within the whole environment leading to the generation of a trajectory j = [ 1; 2; : : : ; Nj ] that contains Nj 3D points. Hence, k represents the 3D information associated at time tk. The number of points Nj depends on the duration of the time interval in which the speci c user Uj is tracked by the proposed algorithm.
Moreover, each trajectory has been t on a smoothing spline to obtain a single continuous trajectory starting from multiple sub-trajectories acquired from each sensor. is a curve de ned starting from a smoothing parameter s (in our experiments s = 0:99) so that the following quantity gets minimized: s X( (tk) i (tk))2 + (1 s)
Z d2 ( dt2 )2dt where tk represents the time in which a point is observed or interpolated. Both ( ) and ( ) are referred to the same time basis. 2.2
Trajectories can be seen as raw data that need to be managed by the classi er to understand whether a behavior of the selected user is anomalous. This goal can be achieved by creating a more discriminative representation of the trajectories, i.e. feature vectors. In this case, eleven features have been identi ed for each trajectory and have been used to de ne the feature vector x = [x1; x2; : : : ; x11] that will be the input of the subsequent classi er. x is populated in the following manner: { the rst ve elements are, respectively: mean, median, standard deviation, median absolute deviation (MAD) and maximum value of the velocity computed on i (de ned as the ratio of the di erence of position on the XY plane and the temporal di erence between subsequent frames); { the next ve elements are: mean, median, standard deviation, MAD and maximum value of curvatures that have been evaluated on the spline trajectory i. Each curvature is de ned as the reciprocal of the radius of the circumference that passes through three consecutive trajectory points; { the last element of the feature vector is the number of trajectory intersections with itself.
It is worth pointing out why the last feature is important for classi cation tasks involving wandering or loitering detection. In these cases, the e ect of moving without any aim or purpose often result in trajectories where loops (or self intersections) are more frequent than when a person travels from a point to another with a speci c goal in mind. 2.3
Three di erent supervised classi ers have been employed in this experiment: a
support vector machine (SVM)[
The next sub-sections will introduce the actual setup used in our experiments, describing the implemented sensors and the system architecture. Input dataset will be presented together with classi cation results obtained by SVM and k-NN and Neural Network. 3.1
Unfortunately, to the best of our knowledge, no public dataset exhibiting walking behaviour of dementia patients has been plublished yet, therefore the proposed methodology has been applied to the analysis of videos produced by a set of RGB-D camera placed within an indoor environment, namely a corridor of a research lab where a group of researchers agreed to take part in the experiment, tasked sometimes of achieving a particular goal (reaching a location, collecting a print, talking, etc.) while in other cases they were instructed of just loiter freely, in order to simulate the abnormal behavior. With reference to the sketch map in Fig. 2a, three Microsoft Kinect sensors K1, K2 and K3 are arranged within the corridor. Speci cally, K2 and K3 focus on the boundaries of the corridor, while K1 looks at the central area (Fig. 2b). Each sensor is locally connected to a node for the data storage, whereas the whole system is remotely controlled by a server unit via a UDP protocol. The server sends a start signal to every node which enable video recording. Each video, which lasts 30 seconds, is nally downloaded by the server. A start signal is sent to the nodes through the network and thus is received with slight delays. However this is negligible and does not a ect the whole system pipeline.
As an example, a frame captured by the Kinect K2 is displayed in Figs. 2c-d, where a depth map and the corresponding RGB image are shown, respectively.
The position of the three Kinect cameras has been set in order to cover the highest area without overlapping of their cones of sight (red regions in the sketch). It ensures the best working conditions for the sensors, since no interference phenomena would alter the depth maps. However, it produces shadow areas, e.g. the regions between K1 and K2 (narrow shadow) or between K1 and K3 (wide shadow). As previously stated, the calibration phase needs a few points of known position in the reference systems of both the kinect cameras and an external surveying instrument.
In gure 2c-d the red circles enclose corresponding objects between the depth and the RGB images, which are used to calibrate sensors and transform data into a global reference system. In order to measure the position and attitude of each camera a theodolite (Nikon Total Station D50[15]) has been used.
In this paper, in order to detect and segment the people silhouettes we have used the well known OpenNI framework together with the Primesense NiTE library[16], which is able to recognize and track up to 15 user skeletons. Although, this framework also integrates a robust algorithm for tracking, it has been used only for the extraction of the skeletal joints, speci cally the torso joint, which is assumed as the center of mass of the detected user. Additionally, users can perform complex movements subtending di erent cameras. Since each Kinect works independently from the others, it is necessary to address people re-identi cation on a higher level, which is not provided by OpenNI.
Two examples of acquired trajectories belonging to the two di erent classes of behavior (normal and anomalous) are reported in Fig. 3, with a single user moving with-in the environment. Here, blue lines display the actual trajectories captured by the RGB-D cameras, whereas the red ones are those generated by spline interpolation, which is also able to reconstruct the user movements out of the elds of view of the three Kinect sensors. 3.2
The preliminary task of trajectory extraction has been used to create a dataset of 60 user paths within the corridor under inspection. Each path refers to the observation of a single individual that has been recognized among the three Fig. 3. Comparison of trajectories belonging to (a) normal and (b) anomalous behavior classes. The inset of (b) highlights the nal part of an anomalous trajectory. Kinect sensors. It should be noted that user paths are extracted also when many people moves simultaneusly in the scene, as the tracking procedure based on the Kalman lter prediction is able to disambiguate the great majority of people intersection.
In the whole dataset, 33 trajectories are associated with a normal behavior and are labeled with 0 (the 55%), while the remaining 27 anomalies are associated to the value 1 (the 45%). A k-fold cross validation method (with k = 5) has been employed to evaluate the capabilities of all the classi ers on the entire available data, since training and test set change and span the whole dataset. For this reason, data has been randomly partitioned in 5 subsets to build the training set with 80% of data and the test set with the remaining 20%. Then, training and testing tasks are repeated 5 times per run, iteratively changing the test set with one of the partitioned subsets. Moreover, in order to better evaluate the accuracy of the tested classi ers, the experiment has been repeated three times per each classi er by changing the initial condition (random seed) used for partitioning.
Results are reported in Table 1. Three experiments are repeated for each classi er, for a total of 9 confusion matrices. The last column reports the average accuracy value for the three runs. The accuracy of each run has been shown in green, under the confusion matrices. The rst thing to notice is that both SVM and neural network are able to exceed 90% accuracy value, implying that the chosen features show acceptable discriminating capabilities when used with such classi ers. On the contrary, K-NN has the worst performances among the classi ers. In particular, the neural network seems to be the best classi er among those considered, as it produces results on average around the 93:9%.
Some examples of the classi ed paths are reported in gure 4. On the left three normal behaviors correctly classi ed, while on the right three anomalous behaviors, which are characterized by repeated changes of the directions or long periods of standing still. In this paper we propose the use of multiple Kinect cameras for developing a low cost system able to recognize anomalous human behavior both for surveillance and AAL applications. The torso node is extracted from the skeleton features provided by the OpenNi framework. A proper Kalman lter is used for the prediction step and allows robust people tracking both inter-camera and intracamera. Anomalous behavior detection is performed using di erent classi cation algorithms by comparing multiple techniques: ANN, SVM and k-NN.
Experimental results demonstrate that the proposed architecture and the developed methodologies are able to recognize anomalous behavior in the majority of cases with respect to the total of observed path. However, it should be noted that the initial association of the paths in the dataset to normal and anomalous behaviors has been done by a human operator observing each path performed by the users. In future researches, more paths will be considered simultaneously to make a decision about a behavior, as those considered as anomalies could be only due to interactions among people.
In its actual form, the system could fail when multiple people enter simultaneously in the camera eld of view and do not maintain the same walking direction while crossing occluded areas. In this case a people re-identi cation procedure [20], based on color features, could be used to avoid false associations and perform correctly the trajectory reconstruction.
Another aspect that is deemed to be important, especially in AAL applications, is the possibility to temporally compare the various observations, for example for detecting medium/long term behavior changes and provide early \warnings" that needs to be noti ed to a caregiver, in order to enable a prompt and more e ective response. Moreover, the compared methodologies have been tested on a dataset with windows of observations limited to a xed length of 30 seconds. It will be worth investigating how other solutions exploting an internal memory, such as recurrent neural networks, can help in detecting anomalies in sequences of arbitrary lengths.
Last but not least, more joints in the skeletal reconstruction will be considered, both for enhancing system robustness and for extending the applicability of the methodology to gait analysis and other applications in the AAL context. 9. Fix, E., Hodges Jr, J.L.: Discriminatory analysis-nonparametric discrimination: consistency properties. Tech. rep., DTIC Document (1951) 10. Haykin, S., Network, N.: A comprehensive foundation. Neural Networks 2(2004) (2004) 11. Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 34(3), 334{352 (2004) 12. Kwon, B., Kim, D., Kim, J., Lee, I., Kim, J., Oh, H., Kim, H., Lee, S.: Implementation of Human Action Recognition System Using Multiple Kinect Sensors, pp. 334{343. Springer International Publishing, Cham (2015) 13. Lutz, W., Sanderson, W., Scherbov, S.: The coming acceleration of global population ageing. Nature 451(7179), 716 (2008) 14. Nie, W., Liu, A., Su, Y.: Multiple person tracking by spatiotemporal tracklet association. In: Advanced Video and Signal-Based Surveillance (AVSS), 2012 IEEE Ninth International Conference on. pp. 481{486. IEEE (2012) 15. Nikon: Total station, http://www.nikon.com/about/technology/life/others/surveying/ 16. OpenNI: Openni website, http://openni.ru/ 17. Piciarelli, C., Micheloni, C., Foresti, G.L.: Trajectory-based anomalous event detection. IEEE Transactions on Circuits and Systems for video Technology 18(11), 1544{1554 (2008) 18. Rashidi, P., Mihailidis, A.: A survey on ambient-assisted living tools for older adults. IEEE journal of biomedical and health informatics 17(3), 579{590 (2013) 19. Reno, V., Marani, R., Nitti, M., Mosca, N., D'Orazio, T., Stella, E.: A powerlinetuned camera trigger for ac illumination ickering reduction. IEEE Embedded Systems Letters PP(99), 1{1 (2017) 20. Reno, V., Politi, T., D'Orazio, T., Cardellicchio, A.: An human perceptive model for person re-identi cation. In: VISAPP 2015. pp. 638{643. SCITEPRESS (2015) 21. Satta, R., Pala, F., Fumera, G., Roli, F.: Real-time appearance-based person reidenti cation over multiple kinecttm cameras. In: VISAPP (2). pp. 407{410 (2013) 22. Shu, G., Dehghan, A., Oreifej, O., Hand, E., Shah, M.: Part-based multiple-person tracking with partial occlusion handling. In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. pp. 1815{1821. IEEE (2012) 23. Warren, A., Rosenblatt, A., Lyketsos, C.G.: Wandering behaviour in communityresiding persons with dementia. Int. J. Geriat. Psychiatry 14, 272{279 (1999) 24. Zweifel, P., Felder, S., Meiers, M.: Ageing of population and health care expenditure: a red herring? Health economics 8(6), 485{496 (1999)