This paper presents our experience on a real case of applying a Wi-Fi ngerprinting-based indoor localization system for monitoring elder people in their own homes. The presented system is part of a broad project called Senior Monitoring where the main aim is to monitor elders to study behavioural patterns as a tool for early detection of some cognitive decay diseases. Since the system is used by real users, there are many situations that cannot be controlled by system developers and can be a source of errors. This paper presents some of the problems arisen when real non-expert users use localization systems, and discuss some strategies to deal with such situations.
In the last years, many Wi-Fi ngerprinting-based approaches have been
presented [
This paper presents our experience in the development of a Wi-Fi ngerprintingbased indoor positioning method to be used for monitoring people at their own houses. In particular, this indoor localization method is part of a bigger project called Senior Monitoring, where the main aim is to monitor elders to study behavioural patterns as a tool for early detection of some degenerative disease such as Alzheimer.
The experiments presented in this paper were conducted in real scenarios and by older adults, where the conditions are not controlled by the researchers. For instance, each user has to create the radio map of his/her own house and, therefore, there is no way of knowing if the training process has been performed correctly. We should trust in the ability of the users performing this task.
In order to introduce a mechanism to obtain a higher degree of certainty about the quality of the training and, therefore, to know if the indoor localization algorithm is performing well, a noti cation system has been developed to collect validation samples. When the system detects stationary moments, the users are asked about where they were at this particular interval of time. The answer is then used to label the samples captured during this period. These labelled samples can be used to validate the accuracy of the system and therefore can be also used to improve the positioning system. But, since people using the application are real users, many non-desirable situations can happen. The user may not be sure of his/her localization during that interval of time, but still answer the question, introducing a possible source of error. Situations like this can produce badly labelled samples that can reduce the accuracy of the classi er.
This paper presents the experience obtained after the system was used by 17 volunteers (mostly elders) for two months on average at their own homes. Users performed the radio map capturing task on their own and used the system following the instructions provided by system's developers. The present work contributes to a better understanding of the di culties and problems that arise when implementing an indoor positioning system in real scenarios with real users. 2
The Received Signal Strength Indicator (RSSI) ngerprinting localization approach requires two phases of operation: a training phase, also known as o -line or survey phase, and a positioning phase, sometimes referred to as online, operational or test phase. In the training phase, multi-dimensional vectors of RSSI values (the ngerprints) are collected and associated with known locations. These measurements are used to build the training data set (also known as radio map) that covers the area of interest. Later, during the positioning (or test/on-line) phase, an RSSI vector collected by a device is compared with the stored data to generate an estimation of its position (see Figure 1a).
In this paper, the position has one dimension, since only the label of the room is used. In this case, the localization problem can be solved using a pattern recognition classi er, where the features and the labels of the training set are the training ngerprints and the locations (room identi ers) where they were captured, respectively. Therefore, the problem to solve in the positioning phase is to estimate the location (room) of the user given the test ngerprint captured at an unknown position. 3
The Indoor Positioning System designed to perform these experiments is part of the research project Senior Monitoring, which is aimed to provide solutions for monitoring elderly people behavior and to detect short-term issues (falls), and long-term issues also (cognitive decay). The IPS consists of a smart-watch, which is worn by the user who is being monitored, and a paired smart-phone, which is used to con gure and control the smart-watch behavior and to communicate with a central cloud server (see Fig. 1b). The server stores the sensory data gathered through the smart-watch and o ers assistance to provide decision support services by performing analysis tasks such as indoor positioning, activity recognition or anomaly detection.
The software has been designed to make use of the following sensors: { Wi-Fi. This sensor constitutes the base of the positioning system. The smart-watch performs a given number of consecutive Wi-Fi scans every minute. The default number of scans is 5, but it can be modi ed. { Signi cant motion. It is a virtual sensor that uses the physical accelerometer, but only triggers when it detects a motion that might lead to a change in the user's location. Thus, though this sensor does not allow to determine the activity the user is performing, it provides a way to detect a possible change in his/her location. Inversely, if it has not been triggered during an interval of time, it may be assumed that the user has not changed his/her location during that period.
Since the system is deployed in home, users manually create the radio map wearing the smart-watch and following the indications of the smart-phone application. The software guides the users to collect the training data in certain points, such as the center or any regularly used location, of each room. When this process nishes, the collected training data is sent to the server. During the system's normal operation, the data acquired by the device' sensors is sent every minute to the paired smart-phone, that dispatches it to the server to be stored and analyzed.
For training, each participant must use the Senior Monitoring smart phone application to create the radio map of his/her home. Users must select the room and the type of training session to be performed. Then, the application communicates with the smart watch and it starts to capture ngerprinting samples. It takes 100 consecutive samples at each training session.
The mobile phone application sends the information to the server, which stores all the samples of each training session, together with information about the user who performed it, date, type of training session selected and the room type.
With this information, the system trains a pattern recognition classi er for each user. This classi er will predict the room type given a set of several consecutive ngerprints as input. A well-known Random Forest (RF) model (using a voting scheme) is used.
Once the classi er has been trained, the application starts to record a sample each minute. Five consecutive ngerprints are captured for each sample. They are sent to the server where the previously trained classi er is used to estimate the current localization (room) of the user. The number of consecutive samples and the capturing frequency can be adjusted through the smart phone application.
A validation phase has been introduced to obtain ground truth information that can be used to get an estimation of the accuracy of the localization system.
For this purpose, the application uses the signi cant motion sensor to look for time periods where the user did not move from the same room. A 20 minutes threshold is used. When a period with no signi cant movement is found, the application sends a noti cation to the user showing the interval of time and a list of the most probable rooms where the user stayed, being the most probable the rst one. The user should select the room that he/she remembers to have been staying during this time period. 5
In order to test our proposed system, 17 volunteers used the Senior Monitoring application for several weeks. Table 1 shows the ID of each user, his/her age, the number of Wireless Access Point (WAPs) that can be perceived in their homes, number of rooms trained, total number training sessions performed (Tr.) and number of answered noti cations (Not.). The last two rows show the median and the standard deviation. Note that most of the users are more than 60 years old.
In order to obtain a measure of the accuracy of the IPS for each user, two ways of estimating the accuracy have been implemented. The rst one, called Noti cation accuracy is the average of times that the user selected the room that the localization algorithm selected as the most probable. Assuming that the user always provides the correct answer, this measure should be a good indication of the accuracy of the indoor localization system. But, since there is not a way of knowing if the users did the task correctly a second way of estimating the accuracy has been also tested. It is called Test accuracy. In this case, all the samples belonging to the time periods of the answered noti cations are labelled using the room label provided by the user as ground truth and then, the classi er estimates the label and compares it with the one provided by the user to estimate the accuracy. Note that in both cases, we are assuming that the user provides reliable information.
The last two columns of the Table 1 show both accuracy measures. It shows that the localization system is working very well for some users, such as for instance the users 1, 3 and 16, but very bad for others, such as users 4, 14 and 17. There are some surprising results as for instance users 2 and 7 that obtain quite better results in the test accuracy than in the noti cation one and users 8 and 9 where the opposite happens. 6
There are many di erent scenarios according to the number of WAPs that can be perceived. Figure 2a shows that there is a relationship between the number of WAPs and the two accuracy measures estimated. The blue points and the blue line are related to the noti cation accuracy and the red crosses and red line are related to the test accuracy. Pearson's correlation coe cients are 0.48 and 0.71, respectively. Figure 2a and Table 1 clearly show that the users 7 and 15 do not follow the regression line, i.e. they are scenarios with a big number the WAPS but obtaining low accuracy. The Pearson's correlation coe cients have been recalculated without the data for these two users obtaining 0.86 and 0.88 showing that the most of cases follow the rule that when more WAPs are perceived in the scenario, better accuracy can be obtained. According to this fact, it is not surprising the bad results of users 12 and 14, with just 1 and 3 perceived WAPs, respectively.
Figure 2b shows the relationship between the number of training sessions and the two accuracy measures estimated. Pearson's correlation coe cients are 0.21 and 0.27, respectively, showing a very weak relationship. There are two users (user 4 and 11) that despite the fact that they performed several training sessions, their estimated accuracy is quite low. Pearson's correlation coe cients without these two users are 0.65 and 0.68, respectively. Therefore, it seems that better accuracy values could be obtained in most of the cases if users had performed more training sessions. In particular, this could be crucial for the users having performed just one training session of each room, i.e. for users 5, 6, 7, 10, 15 and 17.
Figure 2c shows the relationship between the silhouette measure[
Some conclusions that can be derived from the previous discussion is that a good scenario should have many WAPS, a large number of training sessions and the training samples should be well separated. In these conditions, it is expected to obtain good accuracy results. But even in those cases, there are exceptions, i.e. scenarios where the conditions seem to be quite good but the accuracy obtained does not con rm this fact. Some possible sources of problems may be the following: { The Signi cant Motion Sensor used to determine static time periods sometimes needs some signi cant time to detect user motion. Therefore, it is possible that the user moves for some small period to other rooms and comes back to the original position (for instance to go to the bathroom). The system does not detect this movement and throws a noti cation to the user. When the user labels the samples belonging to this time period, the samples belonging to the room where the user stood just for a while are incorrectly labelled. { Sometimes the user does not remember well the place where he/she stayed.
But instead of using the "I do not remember" option, he/she select the one that thinks is the correct one since he/she wants to collaborate with the study and thinks that this is the correct way of behaving. This introduces incorrectly labelled samples. { Users are high motivated volunteers but not always behave as they should.
Sometimes they do not perform correctly the training sessions since they do not understand that there are many ways of introducing signi cant errors in the training set. { There are some special cases that are very di cult to deal with. For instance, in many cases the bathroom is inside the bedroom or in other ones, the living room and the dining room are the same room. These special cases introduce samples very similar in the feature space but with di erent labels. This fact impacts negatively on the performance of the classi er.
This paper has presented our experience on a real case of applying a Wi-Fi ngerprinting-based indoor localization system for monitoring of elder people in their own homes. Since the system is used by real users, there are many situations that cannot be controlled by system developers and that can be a source of errors. From the results obtained, it seems that a good scenario should have many WAPs, a large number of training sessions and the training samples should be well separated. Under these conditions, good accuracy results can be obtained. But even in those cases, there are exceptions. This paper has discussed the possible causes of such low accuracy results. Future work will be focused on improving the estimation of the static time period of the users and to improve the information provided to the users about how to perform the training sessions.
This paper has been partially funded by projects: UJI-B2017-45 and RTI2018095168-B-C53. The authors would like to thank the volunteers for helping us in the developing of the Senior Monitoring project.