An important basis of many smart city applications is knowledge about the location of persons and objects. In outdoor environments, this knowledge can be acquired reliably on a global scale using the wellknown Global Positioning System (GPS). In indoor environments, both the availability and reliability of GPS is signi cantly limited. This, in turn, has led to active research on approaches and systems to enable indoor localization using various cooperating objects technologies. A key challenge during the development of any of these indoor localization approaches and systems is the systematic evaluation of their performance. To do this, developers have to perform extensive and time-consuming measurements at di erent locations over an extended period of time. In this paper, we discuss how the time requirements can be reduced by means of automation. Furthermore, based on our experiences with both, manual and automatic evaluation, we discuss the achievable bene ts and possible limitations.
An important basis of many smart city applications is knowledge about the
location of persons and objects. In outdoor environments, this knowledge can
be acquired reliably on a global scale using the well-known Global Positioning
System (GPS) [
As discussed in depth in [
Especially, when considering the popular category of systems based on scene analysis, a key challenge during the development is the systematic evaluation of their performance. To do this, developers have to perform extensive and timeconsuming measurements in order to calibrate and test the system. Typically, this involves the repeated positioning of objects at di erent locations. In addition, to gather meaningful metrics, it is often necessary to repeat the process over an extended period of time. Consequently, researchers have used simpli ed processes to study the performance of their systems. However, this limits the insights that can be gathered from their results.
In this paper, we discuss how the time required to evaluate the performance
of RF-based indoor localization systems can be reduced by means of automation.
To do this, we outline the typical evaluation process for such systems in Section
2. Thereafter, in Section 3, we introduce two localization systems that we have
built and evaluated as part of WebDA [
Typically, the design of an RF-based indoor localization system starts with the
selection of an appropriate RF technology and a suitable localization approach.
In the past, researchers have studied a multitude of technologies including
mainstream communication technologies such as FM-Radio [
To implement this approach for a given technology, it is necessary to design and ne-tune signal aggregation and comparison functions based on the underlying hardware characteristics. This involves the collection of base measurements, for example, to determine the radiation pattern of antennas, the impact of distance between a sender and receiver on the signal characteristics or the dampening factors of certain materials. Once the design and ne-tuning is complete, it is necessary to evaluate the performance of the resulting system. Again, this involves the collection of measurements using the chosen hardware and the computation of the desired performance metrics on top of the tuned algorithms.
During both, the initial measurements for ne-tuning as well as the nal measurements for evaluation, the following four factors play an important role: { Number of positions: Since approaches based on scene analysis can degrade based on the complexity of the underlying scene, it is usually necessary to collect measurements for a large number of di erent positions. This ensures that the initial algorithm design and tuning is not hampered by speci c e ects of the test environment. Furthermore, it ensures that the nal evaluation results provide a strong indication of the system's achievable performance. { Number of samples: Since most RF technologies are not able to measure signal characteristics perfectly, it is usually necessary to collect multiple samples for each position. This ensures that both the tuning as well as the nal evaluation are not signi cantly distorted through measurement outliers. { Positioning precision: Since scene analysis requires a training phase in which the system is adapted to a particular scene, it is usually necessary to perform 2 consecutive measurements, i.e. one for training and one for validation. When performing these measurements, it is necessary to reposition the signal sources and sinks precisely at the same location twice. Intuitively, in order to ensure that the quality of the data is high, it is necessary to ensure that the repositioning is accurate. { Experiment time frame: Last but not least, it is usually not advisable to perform the training and validation measurements in a short time frame. Instead, it is necessary to stretch out the experiments over a longer period of time. This ensures that temporal e ects, e.g. signal uctuations due to temperature or humidity changes, are represented adequately in the collected measurements.
When trying to minimize the impact of all four factors, it becomes apparent that in many cases, both, the design as well as the evaluation of an RF-based indoor localization system require a considerable amount of precise measurements. Consequently, the measurement process consumes a signi cant part of the system developer's time. While it may seem possible to reduce the time consumption by not considering some of the factors, the impact of doing so is hard to judge and may complicate the design, e.g. due to non-representative measurements, or worsen the expressiveness of the performance evaluation, e.g. due to highly variable and non-reproducible results. In this paper, we argue that it is preferable to reduce the time required by the developer by automating the collection process instead. To clarify this, we present two localization systems that we have built as part of WebDA, one of our ambient assisted living projects, in the following. Thereafter, we describe our experiences with the manual as well as the automatic evaluation of the systems. 3
The goal of the WebDA [
Obviously, both services require accurate and detailed location information. To gather this information in an unobtrusive and automated fashion, we have been working on the development of two low-cost localization systems that rely on active and passive RFID technology. In the following, we brie y describe the requirements on these systems, the details on the hard- and software that we used to realized them as well as their integration with the remaining software. In the next section, we describe our evaluation experiences. 3.1
The two primary application areas of indoor localization in WebDA are the realtime tracking of the elderly persons as well as the on-demand localization of objects. When putting these application areas in the context of ambient assisted living, we can identify the following general requirements which are also often targeted by other systems in this application domain: { Low cost: To be suitable for many elderly persons, the localization system must be inexpensive to deploy and operate. Beyond the cost of the hardware, this also includes factors such as installation cost, the con guration cost as well as the cost for maintaining both, the hard- and the software con guration over an extended period of time. { Low obtrusiveness: To increase the acceptance of the overall system, the localization system should be as unobtrusive as possible. This has a major impact on the hardware form factors that can be used. Moreover, it also has an impact on the actual technology that is used to enable localization, especially when dealing with less technically versed persons. { High accuracy: In order to be usable for the intended purposes, the localization system must be highly accurate. Due to the structure of typical home environments achieving room-level accuracy will not be su cient for many scenarios. As a simple example, consider that in order to return useful results, the object nder service should be able to di erentiate between di erent locations such as "the shelf" or "the co ee table" even if they are in a single room. { High reliability: Last but not least, the localization system must also be reliable. This does not merely refer to reliability with respect to the localization results, but also implies high reliability with respect to small short-term changes in the environment. Such changes may include the movement of furniture such as chairs or the opening and closing of windows, for example. 3.2
In order to support the localization of persons while ful lling the requirements described above, we chose to use an active RFID system that consists of batterypowered tags and readers. The reason for this choice is the fact that when compared to passive RFID systems, active systems typically exhibit a signi cantly higher range and thus, this technology requires a lower number of readers. As the readers are typically more expensive than the tags and we are aiming at localizing a single person with dementia, this reduces the overall hardware cost.
To support the localization of objects, however, we chose to use a passive RFID system that consists of passive tags which are powered through an active reader. The rationale for this decision is that passive tags are signi cantly smaller and cheaper than active tags. Consequently, it is possible to tag a large number of objects of di erent sizes with little additional cost. Furthermore, due to the fact that passive RFID tags receive their energy through the reader, there is no need to replace or recharge the batteries of the tags that are attached to objects.
Figure 1 shows a picture of the main components of the active as well as the passive RFID system. Both systems consist exclusively of comparatively low-cost o -the-shelf components. In the following, we brie y describe each of the main components of these systems in more detail. (a) passive antenna (b) passive tag (c) active tag
(d) active antenna
{ Active RFID System: For person localization, we use the LogiSphere system
from Sensite Solutions [
The localization approach used for persons and objects in WebDA is based on
scene analysis. To be more speci c, we use a ngerprinting technique that is
similar to RADAR [
In general, ngerprinting-based localization is done in two phases that are depicted in Figure 2. During the rst phase the so-called training phase, measurements are made at di erent locations in the target environment. These measurements are then processed - resulting in so-called ngerprints - and the ngerprints are stored together with their respective locations. During the second phase the actual localization phase, runtime measurements are made and these are then compared to the ngerprints made during the training phase. The location of the persons or objects is then determined by choosing the location of the most similar ngerprint captured during the training phase. Thus, using the environment-speci c calibration done in the training phase, it is possible to account for the variances induced by multi-path e ects.
However, in order for this approach to work in our setting using RFID technology for person and object localization, we introduce the following modi cations to traditional ngerprinting approaches: { Time-based aggregation: As the RSS values computed by our RFID readers exhibit a considerable variance (due to measurement imprecisions) we aggregate several readings over time. To account for outliers, we use the 80th percentile measurement as the true value for the interval. To be more speci c, we aggregate readings in a 20 second interval as we experimentally determined this to be a suitable trade-o between accuracy and latency. Intuitively, for the real-time tracking of persons, 20 seconds of latency is already high. To further reduce this, we use a sliding window during the localization phase which typically results in a latency of about 10 seconds.
This latency can be tolerated by our services.
{ Measurement comparisons: To compare measurements of the localization
phase with the measurements made during the training phase, RADAR
proposes the use of the Euclidean distance. To do this, the readings of a single
tag made by each reader are represented in a vector Rk where k denotes
one of the readers. We rely on the same basic approach. However, to reduce
the number of comparisons, RADAR proposes to further aggregate
multiple measurements made at the same location using the component-wise
maximum. Since we did not nd this to be necessary in order to achieve a
satisfying performance, we are refraining from using this step.
{ Multiple tags: In the past, researchers have showed that by introducing more
readers, it is possible to improve the resulting localization accuracy.
However, as the readers are typically much more expensive than tags, we are
proposing to use multiple tags instead. In [
In order to make the previously described localization hardware and algorithms usable by the user-facing services implemented in WebDA, we have integrated them. Furthermore,to speed up the adaptation of these services to di erent home environments, we have developed a number of tools. In the following, we brie y describe the resulting architecture as well as the provided tool support. Architecture The web-based services implemented by the WebDA project rely on the OSGI component model for modularization. Consequently, we used the same framework to implement the localization system. The building blocks of the resulting localization system are depicted in Figure 3a.
(a) Architecture (b) Tools
At the lowest level, a number of drivers provide access to the active and passive RFID systems. On top of that, the localization algorithm is implemented as an OSGI service. Using this service, other services can subscribe to localization events which indicate the location of persons and objects. Furthermore, they can trigger passive scans in order to start searching for objects. By subscribing to all localization events, a localization history service can store all location information persistently over time. Furthermore, it can make the history of locations available to other services, for example, to perform long-term pattern recognition or to facilitate testing and debugging. In order to abstract from the concrete geometric properties of a particular home environment and in order to generate user interfaces that are meaningful for the users of the system, the localization events are expressed in terms of a symbolic location model. The location model encompasses relevant "zones" such as rooms and other areas, e.g., closets, tables, etc. Furthermore, it models the geometrical containment relationships between the zones. This enables the user interface related services to display, for example, hierarchical menus and re ned outputs such as "the object is in the living room close to the couch". Intuitively, this model has to be adapted speci cally for each environment as the geometrical properties change between environments. Tools In order to con gure the localization system for a particular home environment, it is necessary to a) model the environment and b) perform the calibration required as part of the training phase of the localization algorithm. To support both tasks, we developed a simple tool on the basis of the Eclipse platform. A screenshot of this tool is depicted in Figure 3b.
Using a graphical editor, a user can model di erent areas which constitute the environment. Based on the Eclipse Graphical Editor Framework, the editor supports all common actions such as insertion, movement by dragging, deletion, etc. and it is also possible to specify exact coordinates by means of an Eclipse Property View. The modeled areas may refer to di erent concepts such as di erent types of rooms or zones within them. By providing an approximate geometry of the areas, the tool can automatically gather the containment relationship. By clicking an export button, the modeled environment can be exported to a database format which is then used by the localization system. In order to calibrate the system, the Eclipse tool can also be used to pinpoint the current location at which a measurement for a ngerprint is taken. The resulting ngerprints and their locations can then be fed into the localization algorithm which can then a) compute the location and b) attach the location to an area in the location model. From our experience, this tool signi cantly speeds up the overall deployment process as the visual representation of areas and locations speeds up the database con guration and reduces the potential for attaching false locations to the measurements during calibration. 4
To evaluate WebDA's person localization system, we have performed an extensive manual evaluation. Furthermore, to evaluate the object localization system, we have recently started with an automated evaluation using our TrainSense testbed. In the following, we describe our experiences during both types of evaluations. Thereafter, we highlight the bene ts and limitations of each approach. 4.1
In order to measure the performance of WebDA's person localization system,
we performed a manual evaluation the results of which are described in [
To test the accuracy of object localization, we have performed further
experiments. However, in order to reduce the time requirements induced by manual
evaluation, we tried to automate the data collection process. As the experiments
required us to place the tags at a number of positions in a repeatable manner, we
decided to use our TrainSense testbed for automation. TrainSense uses
miniaturized model trains to simulate the nomadic movement of cooperating objects. In
the following, we brie y outline this testbed before we describe our experiences.
TrainSense The TrainSense testbed [
Figure 4a illustrates the basic hardware architecture of TrainSense. The system is based on o -the-shelf model trains and it consists of a computer host, a controller, train detectors, tracks and locomotives. This setup is similar to what is commercially available. The tracks, detectors and trains are bought o -the-shelf. The tracks are mounted on wooden plates which can be combined arbitrarily in order to create di erent layouts for di erent experiments. The wiring required among the di erent components of the train control is exactly the same as the one required for standard model trains and it is built into the modules. Similarly, the communication protocols used in our system are standard, to allow the reuse of existing, low-cost hardware components (Maerklin/Motorola I for the train control and S88-N for the train detection).
(a) TrainSense Hardware Architecture (b) Software Architecture
Fig. 4: TrainSense
However, the hardware and software of the controller and the detection mechanism have been re-designed to satisfy the needs of precise and repeatable wireless experiments. The controller is a real-time system, which means it reacts to the perceived events, such as the detection of a train at a particular location, within a bounded amount of time. Beyond providing the ability to detect the passage of the train at a detector, this also enables us to let it run for a precisely determined duration and to stop the train quickly. If we do not consider extremely rare mishaps like stalling or derailment, this allows us to place the train with a guaranteed precision.
For most of our evaluations, we use basic dead reckoning to position the tagcarrying train. For this, the train is set to run at a constant speed. When it crosses a detector, the controller is instructed to let it run for a programmed amount of time (proportional to the distance from the detector to the targeted position). After that time has elapsed, the controller stops the train. Once the train has reached its destination, the host starts the code that performs a measurement for the current position.
The resulting overall software architecture is depicted in Figure 4b. The gure shows the three main components of the testbed, which are the train that is able to move to a position (left), the real-time controller that controls the movement of the train and the host computer (right) that runs the experiment by performing the following steps: 1. Move the train to the rst position and signal the arrival to the software on the host computer. 2. Once the arrival is signaled, start the measurement software and wait for its completion. 3. If there are more positions, go to the next one, signal its arrival and repeat from step 2. 4. Else signal the end of experiment and run the measurement software to compute the results. Experiences For the evaluation of the passive RFID system we were interested in how the radio signals behave over time for certain positions. We therefore set up the experiment with a ve meter long testbed shown in Figure 5 making measurements every meter. This results in six di erent positions marked as P1P6. TrainSense o ers the possibility to send the Trains to xed positions in the testbed with a precision of less than 2 cm. Due to the nature of the track layout this error can also only occur in one dimension. In our experiment this gives a upper bound for misplacements by 2%. For the capturing of the signals we used four antennas connected to the passive RFID reader. Two of those antennas (R1,R2) were placed 50cm from the beginning of the tracks and 50 cm to the side while the other two were placed in a similar fashion on the other side of the tracks. In order to be able to measure the signals of the tags at di erent positions we attached three RFID tags to the train. These tags were arranged in a triangle to expose di erent angles to the readers antennas. The photo in gure 6 shows the real setup as we have taken the measurements.
While setting up the experiment certainly takes a few hours because of the placement of the rails, antennas and other hardware factors this is only needed once before starting the real experiment. After the sensing software is connected with the TrainSense platform and the required positions are programmed, the experiment can run unattended. This is especially handy when things go wrong. So when for example the cable of an antenna is not properly connected it is easy to repeat the whole process. This also allows for repeating the same experiment again later without wasting the system developer's time.
In our case we ran the experiment twice with some hours in between. We then analysed the results of both sets as follows. In each set signals we collected for a period of two minutes per position. We divided each of these sets in 8 consecutive bins resulting in 15 seconds long parts of the data. For each of these sets we calculated the average signal strength and standard deviation for each tag - antenna combination. We found that for these intervals the standard deviation is close to zero in almost all cases so we continued our evaluation only considering the average signal strength. We then did a cross-validation on the 8 sets per position measuring the di erences in the average signal strength. Figure 7 shows the result of this evaluation depicting the number of occurrences of certain signal di erences within the single measurement runs and between the two runs. The graph also depicts the number of sets where one tag was read at one antenna in one set but not in the other set.
It can clearly be seen that when using only one measurement a cross-validation the signal di erences are only minimal and for the majority of the data well below 1 db. While this is true for both single sets the same evaluation using both sets shows very di erent results. Here it can clearly be seen that the majority of di erences are around 3db. When considering the number of sets that have one tag-antenna combination in one bin of the rst set but not in the bin to which it is compared it becomes clear that the likelihood of such an event increases with time. This experiment shows that for the evaluation of ngerprint based systems it is essential to have two disjunct sets of data that are used for training and testing. When such a system is tested with only one data set which is then split into di erent bins for a cross-validation the results may not be accurate. Evaluating localization systems with the TrainSense platform has shown high potential. While it might require some time to set up the testbed and connect the measuring software to the testbed controller the amount of time that can be saved during the actual experiments will pay o very fast. This is especially true for large scale experiments where a lot of di erent positions need to be measured. Running such an experiment manually would block the system developer for the whole time whereas TrainSense makes it possible to run the experiment unattended. This makes TrainSense a good choice for measuring signal-strengths in the case of object localization or when trying to simulate the behaviour of the senders without any interferences { which is a typical measurement during the initial system design. On the other side TrainSense is not capable of simulating dampening factors such as a human body. If such factors are needed for the system it will still be necessary to do the measurements manually. As discussed in detail above this can be a very time-consuming task and therefore it might seam reasonable to use cross-validation on only one dataset to generate the needed results. As shown in Figure 7 this might not lead to reasonable results. 5
Knowledge about the location of persons and objects is an important basis of many smart city applications. The unavailability of GPS in indoor environments has led to active research on approaches and systems to enable indoor localization using various cooperating objects technologies. A key challenge during the development of any of these indoor localization approaches and systems is the systematic evaluation of their performance. In this paper, we described how the time requirements can be reduced by means of automation. Furthermore, based on our experiences with both, manual and automatic evaluation, we discussed the achievable bene ts and possible limitations. Based on this, we argue that in cases where it can be applied, automation is preferable to simpli ed experiments as these may lead to less expressive results. At the present time, we are completing our automated evaluation of the WebDA object localization system using our TrainSense testbed. Simultaneously, we are working on several improvements to the software as well as the hardware of the testbed in order to further increase its applicability to other scenarios.
The work presented in this paper has been partly funded by the German Federal Ministry of Education and Research as part of the WebDA project under grant number 16SV4023 and the European Center for Ubiquitous Technologies and Smart Cities (UBICITEC).