Extended life expectancy in developed countries is a sign of a global better health, but it brings with it needs and challenges to be properly addressed. To this end, Information and Communication Technologies (ICT) can provide solutions for Active Ageing. However, the success of such solutions is strongly affected by the perception of the users about their obtrusiveness and efficacy. In this context, the NESTORE project aims at developing a suite of personalized guidelines for supporting health and global wellness to be provided to the user by means of multimodal tools: a virtual coach, a tangible interface, serious games, and a set of personal and environmental sensors.
One of the main goal of NESTORE is to monitor physiological and behavioural data related to the five human domains of well-being (i.e., physical, mental, cognitive, social, and nutritional), as identified by medical specialists [5]. In order to achieve this goal, we develop a multi-domain unobtrusive monitoring system, also relevant for assessing the user profile [36] [28]. The NESTORE system optimizes and integrates available technological solutions based on advanced non-invasive monitoring systems, such as wearable and environmental sensors.
In this paper, we show the actual devices chosen as the set of environmental sensors of the NESTORE ecosystem and, in particular, the tests performed on custom devices, namely Bluetooth Low Energy (BLE) beacons, developed in the project's activities with the aim of detecting social interactions, indoor user's behaviour, and the environmental status of the user's vital space (indoor air quality, movements, detection of indoor/outdoor).
In recent years, we have witnessed a rapid surge in assisted living technologies helping the ageing population [31]. Recent advancements in several technological areas have helped the vision of Ambient Intelligence (AmI) and Ambient Assisted Living (AAL) to become a reality. These technologies include smart homes, assistive robotics, e-textile, and mobile and wearable sensors. Among the currently available enabling technological solutions, BLE is becoming the most prominent in allowing interoperability of wireless technologies in smart environments [24,38] and it has been widely adopted in different typical AmI/AAL scenarios, from indoor positioning [37,29] to human activity recognition [13] and health status monitoring [35].
In the NESTORE ecosystem, besides the unobtrusiveness granted by this kind of devices (small in dimensions, ease of installation and maintenance, long duration of battery), the choice of BLE allows us to provide different kind of services to the user with the same devices (i.e., social interaction detection, indoor behavioural index, and indoor thermal comfort) using the inherent capabilities offered by this technology. We exploit the characteristics of the BLE technology in terms of type of customizable BLE messages (the BLE devices embed different sensors like accelerometers, humidity and temperature sensors, which information is transmitted in the advertisement's payload) and possibility to infer proximity using the Received Signal Strength of the received BLE advertisements.
The rest of the paper is structured as follows: Section 2 shows the main components of the NESTORE indoor monitoring system with Section 3 describing the actual implementation of the BLE solutions in the task of detecting social interactions, inferring the indoor beahvioural index, and monitoring the indoor thermal comfort of the user's home.. Section 4 illustrates the main tool used as feedback to the user, namely the NESTORE coaching app, and how it is used to show the results of the algorithms presented, while Section 5 draws the conclusions and future works.
The NESTORE environmental monitoring system is an ensemble of wireless sensors able to sense the variables indicated by the domain experts. Furthermore, it has the aim of detecting the interaction of the user with his circle of friends and caregivers and the environment itself, while monitoring the status of the environment, in terms of indoor thermal comfort. For this reasons, we call environmental device any sensor deployed in the user's vital space, while wearable the device worn by the user during his daily activities. As further source of information about the user's status, we derive data as result of computation and fusion strategy from a direct input of the user, as questionnaires, while interacting with the NESTORE coach app. We call the latter soft data.
In order to build the integrated NESTORE environmental monitoring system, we first performed a technology selection to satisfy not only the requirements coming from the domain experts related to the user profile (to cover as much variables as possible), but also the ones coming from the co-design activities in terms of unobtrusiveness. The integrated system should be unobtrusive under diverse perspectives: R1) user interaction -the user should not wear additional sensors or explicitly interact with the environmental device; R2) number of devices -the user's living environment should not be filled with lot of visible devices; R3) installation and maintenance -it should be easy to deploy and maintain the device without additional effort from the user.
In this paper, we focus on the environmental technologies used to cover the variable indicated by the medical experts of NESTORE, in terms of: sleep monitoring [15,2], weight monitoring [6], indoor user's behaviour and environmental status monitoring [10,30,11], and social interaction detection [25,7,14]. Figure 1 shows the ensemble of devices and technologies chosen to build the NESTORE environmental monitoring system. In particular, we will describe the methods that leverage the presence of BLE beacons as primary source of information, being the most suitable solution to address the requirements of unobtrusiveness as listed above.
The BLE beacons are emitters that periodically send beacons (called advertisements in BLE) with a specific rate and power. They are received by other BLE devices located nearby (e.g., the NESTORE wristband). When the receiver hears a beacon, it estimates the Received Signal Strength Indicator (RSSI), a value expressed in a decibel scale (dbm), which can be related to the distance of the receiver from the transmitter.
The hardware kit given to the NESTORE users during the piloting phase is composed of all the environmental devices (e.g., a smart scale, a ballistocardiography sensor, five social BLE beacons, and five environmental BLE beacons) plus the NESTORE wristband with its charger. The subset of devices based on BLE connectivity is reported in Figure 2. It is composed by the wristband, a charging station, and five social and five environmental beacons. The software modules that are going to be described in the following subsections analyse data uploaded to the NESTORE cloud through the NESTORE Connect agent, a service running in background of the user's phone.
This section describes the algorithm to detect the social interactions between the NESTORE user and his/her circle of friends/caregivers, the algorithm is referred to as Social Interaction Detector (SID). The basic idea is to analyse the messages emitted by the social BLE beacons [8] with the goal of detecting proximity among users. In turn, the proximity between a pair of users (a dyad ) is considered a sociological marker that express the willingness to establish a social relationship, as also studied in [3]. Differently from [20], SID has been designed as a Cloud-based service so that to elaborate the data collected remotely and automatically at periodic intervals. Moreover, SID can analyse data stored in different data providers such as a MongoDB collection, a plain CSV file or the NESTORE Robofuse server. SID modifies also the way the interactions are initially detected and maintained with respect the approach followed in [20].
Each NESTORE user wears the wristband acting as a data logger of the messages emitted by the social BLE beacons. Beacons are given to the NESTORE user's local circle: a set of friends/relatives/caregivers/neighbours etc. that the NESTORE user identifies during the installation process.
The whole flow comprises the following steps:
1. Messages emitted by the social and environmental beacons are collected by the wristband. 2. Messages are uploaded through the NESTORE Connect agent to RoboFuse 5in the NESTORE cloud. 3. SID periodically fetches and analyse the data from each NESTORE user. 4. The result of the analysis of SID is uploaded to ZivaCare6 through dedicated APIs. 5. Results are available for a graphical representation with the NESTORE coaching app.
SID is implemented in Java programming language; it runs on a dedicates Virtual Machine (VM) hosted on the NESTORE cloud. SID performs two core operations at periodical intervals: User profiling and Data retrieval and analysis.
User profiling During the user profiling operation, SID first retrieves the list of NESTORE users and it checks if each of them has granted the consent for the analysis of data. If the consent is not given or denied, then the analysis of the specific NESTORE user is skipped, otherwise SID retrieves the user's profile:
-The list of IDs of the social beacons assigned to each NESTORE user;
-The list of IDs of the environmental beacons assigned to each NESTORE user; -The list of IDs of the friends joining to the NESTORE user's local circle; -Other meta-information for each NESTORE user.
Data retrieval and analysis In this phase, SID retrieves for each NESTORE user the messages collected from its own wristband. Then, it performs the analysis of the messages collected from the wristbands of each NESTORE user. SID retrieves messages for a time period (e.g. the last 24 hours) with the goal of identifying the number and the duration of the social interactions of the NESTORE user with his/her local circle's members.
The algorithm considers an interaction between X (NESTORE user) and Y (NESTORE user's friends) composed by 3 stages: i) Opening; ii) Keeping; iii) Closing.
The opening stage lasts for ∆ up seconds during which SID checks two conditions:
-Number : to receive at least p % of the expected messages. The expected messages depend on amount of messages a wristband can collects in a time period; -Quality: the RSSI of the messages received must exceed a specific threshold t.
If both of the two conditions hold, then SID starts an interaction between X and Y . The keeping condition lasts for an undetermined period of time. During this stage, SID checks the Quality condition. In particular, the RSSI of the messages must always exceed a specific threshold t.
The closing condition checks that for at least ∆ down seconds no messages holding the Quality condition are received. If this happens, then SID closes the interaction between X and Y . A simplified representation of SID is reported in Figure 3. The figure shows a time interval split in three segments: i) ∆ up , during which SID expects at least the p % of beacons with RSSI over a threshold (green dots); ii) the keeping segment; and iii) the ∆ down segment. SID allows to identify not only the number of interaction for a dyad (X-Y ), but also its duration. In fact, all the messages analysed are annotated with a timestamp which is used to keep track of the first receives message (starting time) and the last receives message (ending time).
Data collected with SID can be used to estimate the location of an interaction. Such information is obtained by analysing messages coming from the environmental beacons. Environmental beacons are similar to the social beacons with the difference that they are deployed in specific locations of the house, e.g. kitchen, bed room, rest room etc. The basic idea for detecting the location of the interactions is similar to the algorithm for detecting interaction among people: if SID detects proximity between NESTORE user X and environmental beacon E 1 deployed in the kitchen at time [t 1 − t 2 ] and if SID detects proximity between NESTORE user X and user Y during [t 2 − t 3 ], then SID infers that NESTORE user X interacts with Y in position E 1 from time t 2 to time t 3 .
Performance evaluation In order to test and calibrate SID with a suitable configuration for the NESTORE pilot studies, the following tests have been conducted with the goal of: i) calibrating SID for detecting the social interactions (parameters p and t); ii) measuring the performance obtained by SID.
The first test (T est1) reproduces a social interaction during which the friend always brings the social beacon in proximity: the NESTORE user wears the wristband and the local circle is composed by one person (the social beacon is configured with a power of emission of −8dbm). The protocol for reproducing the social interaction is the following (six runs with the same protocol):
1. The dyad moves 15+ meters away, in order to reproduce absence of interaction: (a) 4 minutes of non-interaction. 2. The dyad moves 1 to 1.5 meters in proximity, in order to reproduce interaction: (a) the friend brings the social beacon on the key-chain; (b) 4 minutes of interaction.
The second test (T est2) reproduces a social interaction during which the friend "forgets" the social beacon far from the place where the interaction actually takes place. This scenario reproduces a common situation in which the a friend visits a NESTORE users and it leaves the bag at the entrance of the house: the NESTORE user wears the wristband and the local circle is composed by 1 person (the social beacon is configured with power of emission: -8dbm/ -4dbm / 0dbm. In this case, we tested different settings of the social beacons). The protocol for reproducing the social interactions is the following (6 runs with the same protocol): We measure the performance of SID during the interaction and non-interaction scenarios of T est1 and T est2. To this purpose, we compare the results of SID with respect to the ground truth. Figures 5 and 6 report a representative example of the performance of SID. The red bars represent the temporal intervals during which a dyad is actually interacting; the blue bars represent the output of SID. The more the bars overlap, the more SID performs well. In order to quantify the matching between SID and the ground truth, we measure the following metrics: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). These metrics assess the number of right/wrong answers of SID with respect to the number of observations in the ground truth (i.e., interaction or non-interaction for a specific dyad). Given such core metrics, we measure the accuracy as: accuracy = T P + T N T P + T N + F P + F N which assesses the proportion of correct answers of out SID with respect to the total number of observations. We also measure the F1 score, which combines both precision P = T P/(T P + F P ) and recall R = T P/(T P + F N ), as follows:
reports Accuracy and F1 score for T est1, respectively. We show a set of lines of different colours, one for each of the values of selected p and for different values of t (from -95dbm up to -75dbm). Accuracy and F1 score decrease progressively with the increase of the t. In particular, the higher the RSSI threshold used to infer proximity, the more errors are introduced by SID.
Differently from T est1, in which the social beacon is always close to the interaction place, T est2 is more challenging. In particular, SID has to detect a meeting even if the social beacon is far from the place of the interaction. We tested the social beacon with different powers of emission. The tests done with -8dbm (like in T est1) do not result with a positive outcome. In particular, we observe that SID reports too many false negative answers: non-interaction rather than interaction. This is caused by the low power of emission of the messages that cause a high rate of errors. In order to increase the performance also in T est2, we set social beacons to -4dbm and 0dbm. The results of Accuracy and F1 score are reported in Figure 10. In this case, the Accuracy and F1 score have acceptable values up to t = −85dbm. After such threshold, Accuracy and F1 score decrease remarkably.
The objective of this module is unsupervised user habits detection starting from the information coming from the environmental BLE beacons. By monitoring the activation of the sensors embedded in the BLE beacons, indicating the opening/closing of doors and room occupation of the user during his time spent at home, it is possible to retrieve heterogeneous and multivariate time-series over long periods. These time-series can be used to learn recurrent behaviors of the user in her/his daily activities by analysing the time variations of several parameters like the room occupied by the user and qualitative activity level. In the literature, many applications, which serve mainly to support diagnosis, effectively deal with temporal sequences, encouraging the development of the related "time series mining" research field [34,32]. Discovery algorithms aim at extracting important pattern such as similarities, trends, or periodicity, with the aim of recurrent pattern description or prediction [26]. Encouraging results in building behavioral profiles of a person living in a smart home are highlighted in [17] in which a feature mining algorithm is presented. Also in [23] and [22] behavioral pattern identification methods are proposed using binary similarity and dissimilarity measures on data generated from occupancy sensors including door and motion sensors in a smart home. Understanding the behavioral profile of a user is extremely important to detect behavioral changes possibly related to a deterioration of the user physical and psychological status. This is an emerging research topic addressed in several works for supporting independent living of older people. In [21], authors describe a solution based on home automation sensors, including movement sensors and door entry point sensors. By monitoring the sensor data, important information regarding anomalous behavior are identified using supervised approaches to predict the future values of the activities for each sensor in order to inform the caregiver in case anomalous behavior is predicted. Within the scope of the NESTORE project, we intend to address the unsupervised detection of these forms of behavioral anomalies, since collecting ground truth information for a long period in a real house can be very obtrusive for the user. For this reason, we focus on motif search on sensory data collected in pilot sites, represented as time series, by exploiting the results obtained in the field of time series motifs discovery [19,1]. Time series motifs are approximately repeated patterns found within the data. Such motifs have utility for many data mining tasks, including rule-discovery, novelty-detection, summarization and clustering. Since the formalization of the problem and the introduction of efficient linear time algorithms, motif discovery has been successfully applied to many domains, including medicine, motion capture, robotics and meteorology. An emerging technique used in the field of motif discovery is represented by "stigmergy" [30,9]. This is a term derived from the research on the foraging behavior of ants, which communicate with each other exchanging information through the modification of the environment and the information can only be accessed when an ant visits the place marked by another ant. Several works used this technique in order to infer motifs in time series related to different fields, from DNA and biological sequences [12] to indoor localization [29] and intrusion detection systems [16].
In the NESTORE scenario, physical displacements of users in their vital environments can offer information about the change of their individual behavior, capturing all the areas (rooms in the home) visited by the user over time. In this domain, useful insights are given by [21] in the field of the representation of sensor data for further analysis on behavior deviations detection. Authors propose two different techniques for the summarization of data: combined activity of daily living signal as a time series and start time and duration. The first method involves the use of a signal assuming different levels for each activity of daily living, where each level of the combined signal represents one of the sensors triggered by the user. In the second one, the signal is represented by the start time and the duration of an event representing the user entering in a room and the duration that she stays in a specific location. This approach overcomes one of the biggest limitations of existing state-of-the-art techniques in pattern discovery regarding the possibility of discovering pattern occurrences having the same time length, failing to capture similarities when the occurrences are uniformly scaled along the time axis.
In NESTORE, the module in charge of discovering routines and habits of the user is composed of two tasks: one able to tailor the system to the user based on his daily routines and a task that detects days with abnormal behavior, in order to check the effectiveness of the system and to monitor any possible problem in terms of missing engagement of the user or malfunctioning parts of the system. The detection of the indoor behavior of the user uses data coming from BLE environmental sensors. In particular, five environmental BLE beacons are deployed in the user's home to give information about the indoor mobility of the user and his interaction with relevant Point of Interests (PoIs) of the house. During the installation of the NESTORE system, the five BLE beacons are deployed in the most used areas of the house (i.e., kitchen, living room, and bedroom) and on commonly used furniture, like the door of the fridge and the bathroom door. The BLE beacons, leveraging the capabilities offered by the radio propagation of the Bluetooth signal, provide information about proximity of the data-gathering device (i.e., the wristband) worn by the user with the beacon itself, therefore the position of the user in the area in which the beacon is installed. Furthermore, the beacons embed an accelerometer that is activated when it is moved, therefore when the furniture is used. Given this scenario, the module is able to detect the movements of the user inside the house and his behavior in terms of occupied areas and interaction with the PoIs over the long period. In order to provide an indoor behavioral index, the module computes the total occupancy duration over the day for each area. The identified areas are the three rooms in which the beacons are installed and two additional areas: "other indoor", when the system is not able to correctly identify a precise room but collects data indoor, and "outdoor", when no beacons are collected.
The Indoor Behavior Index (IBI) is computed as follows:
where, A i (T ) represents the percentage over the 24 hours of the occupancy duration in the area A i during day T , while A i (T − 1) represents the percentage over the 24 hours of the occupancy duration in the area A i during the previous day (T − 1). Table in Figure 11 shows two weeks of percentages for occupancy duration generated randomly, in order to show how this information can be used to compute the IBI. Once the IBI is computed for each day, the module can estimate if there is an abnormal behavior in a particular day, when the IBI falls outside the range of mean(IBI) over a baseline week plus a threshold of 20%. These values can be parameterised accordingly to the user's baseline behavior. Figure 12 shows the trend of the generated IBI over the two weeks of observation from data shown in Figure 11. We can see that, in this way, it is easy to detect an abnormal behavior during day 8, 9, and 10.
The formula used to compute IBI can be modified to detect changes related to a particular day, for example the same day of the previous week, as follows:
where, A i (T − 7) represents the percentage over the 24 hours of the occupancy duration in the area A i during the same day of the previous week. Once a day with abnormal behavior is detected, the system can check the number of activation of the beacons installed on the fridge and bathroom door in order to further analyse the behavior of the user in terms of interaction with the house. The aim of this module is twofold: (i) the validation of the recommended plans provided by the NESTORE DSS [36,28], especially when dealing with sedentariness; (ii) to check the usage consistency of the provided devices, for example a day with anomalies can be related with a non-properly functioning/not worn wristband or an offline beacon.
Thermal comfort is defined in [27] as "the condition of mind that expresses satisfaction with the thermal environment". Due to individual differences, it is impossible to specify a thermal environment that will satisfy everybody. There will always be a percentage of dissatisfied occupants, but it is possible to specify an environment predicted to be acceptable by a certain percentage of the occupants. This leads to an evaluation by a subjective point of view but over the years, a large amount of empirical studies has been conducted how which parameters are the most influential. These parameters can be divided into personal and environmental factors. Personal factors include clothing and personal activity and condition, while environmental factors comprise thermal radiation, temperature, air speed and humidity. Following the recent trend in literature [33] and the large availability of Internet of Things (IoT) devices on the market, we focus on temperature and humidity as key parameters to identify indoor thermal comfort.
In literature, we can find two main standards specifying indoor thermal comfort: ISO EN 7730 [18] and ASHRAE 55-1992 [4]. Besides complex models that include air speed and thermal radiation (predominant outdoor rather than indoor) and also involve personal factors (e.g., predicted mean vote, clothing insulation, metabolic rate, etc.), both standards propose simpler models for thermal comfort based on Relative Humidity (RH) and Temperature (T), easier to be measured and controlled in indoor environments.
Figure 14 shows the RH/T diagrams proposed by the two standards. The IS EN 7730 approach for defining the comfort zone does not take into account the fact that higher temperatures can be tolerated at low humidity. Hence, its lower and upper temperature limits are vertical. Although temperature ranges are specified per season, the relative humidity is set between 70%RH and 30%RH in summer and winter time, respectively. The limits are set to reduce the risk of eye irritation, dry or wet skin, microbial growth, and respiratory diseases. ASHRAE 55-1992 also considers the effect of experiencing higher temperature together with high relative humidity (as can be seen in Figure 13b, in which graph presents oblique boundaries for the lower and upper limits of T). In the NESTORE scenario, we implemented the ISO EN 7730, to present to the user the thermal comfort status of his house.
One of the main points of the interaction between users and the NESTORE ecosystem in represented by the coaching app. A big role in the developing process of the NESTORE User Interface has been played by the interaction design approach. During the design process, the system is divided into small modules, defining the basic building blocks of the User Interface. Then, the foundational elements are combined to create functioning units with different features. Finally, units are combined in order to form different sections of the interface. This flexible methodology, accommodative to changes, allows creating complex systems, managing consistency and scalability.
Figure 16 shows some screenshots of the NESTORE coaching app regarding the output of the elaborations based on the environmental devices. In particular, we can see the graphs produced for indoor social interactions (Figure 15a) and the indoor status of the user's environment in a good (Figure 15b) and bad (Figure 15c) status, together with the screens of one of the other services based on environmental device, like the sleep monitoring systems (Figure 15d).
Thermal comfort (Good) Thermal comfort (Bad) Sleep monitoring Fig. 16: The charts provided to the NESTORE users.
The proposed systems for detecting social interactions, indoor behavioural index, and indoor thermal comfort take advantage of the BLE beacons capabilities of inferring proximity among devices and in terms of ease of embedding different sensors. They represent the core set of environmental devices composing the NESTORE monitoring system. The BLE beacon capabilities enable rich context-aware interaction scenarios with the NESTORE system. For future works, being able to know in real-time the position of the user, the system can proactively deploy notifications and reminders in the different user interfaces of the NESTORE system. Indeed, proxemics can be used to enable different interactions: when the user is approaching the tangible coach, the vocal assistant could start speaking; when the user is in the same room of the tangible coach, peripheral light (LED in the tangible) can be used to display a notification; when the user is an a different room, sound notifications can be used; when the user is not home, notifications can be sent in the NESTORE Coach app.
This work has been funded by the EU H2020 project "Novel Empowering Solutions and Technologies for Older people to Retain Everyday life activities" (NESTORE), GA769643. The authors wish to thank all the partners involved in the project.