This paper describes an innovative sensor system which can detect and track people in a room by means of an array of capacitive sensors beneath the floor covering. By combining cutting-edge technology from the domains of capacitive sensing, wireless data transmission, interconnecting technology between textiles and microelectronics and high level data processing it is possible to support various groundbreaking applications in the domains of Ambient Assisted Living, energy saving, comfort, marketing, healthcare and security.
The functions of room floors range from mechanical support, convenience, heating
and noise reduction to the expression of individual style and design. However,
considering the fact that during the day we are mostly in direct contact with the floor one
may ask whether it is possible to exploit this close relationship for even more
advanced functions. Within this paper we will demonstrate how a room's floor can be
transformed into a sensor plane that detects and monitors people's behavior and allows
for a collection of novel supportive functions. Although these functions are obvious to
the user, the sensor system itself remains totally invisible and does not interfere with
the material or design of the floor covering in any way. In this respect, the technology
we present here is an example for a new class of systems aimed at applications
summarized under the expression Ambient Assisted Living (AAL) [
The basis for the functions offered by our sensor system is the detection and tracking
of people moving around within the room. Whereas the detection of general
movement can be achieved with cheap infrared or ultrasonic motion sensors, for instance,
the acquisition of people's exact location requires more advanced systems e.g. based
on camera image processing [
Our system relies on a much more direct way: a grid of sensors underneath the floor
covering detects local capacitive changes in the environment brought about by humans
walking on the floor. By design, this method does not allow for an identification of
persons (see [
The capacitive measurement principle allows for a unique advantage of the system compared to pressure sensors: as the sensors react from a certain distance without direct touch, there is no restriction on the floor covering. The system works under carpet, linoleum, laminate, wood and even tiles or stone floors.
In the following section the technology for the capacitive sensor system is described in detail followed by the presentation of the data transmission techniques we have implemented to extract the data from the sensor grid.
Next, an algorithm is presented, which relies on nonlinear dynamical systems to reconstruct the location of a person from the sensor data.
After that, we show which applications can be realized by means of such a sensor system and which requirements are imposed on the sensor data processing by these applications. The paper closes with a summary.
In the following we will describe the system's components for sensor data acquisition, information transmission and processing and their interaction in detail.
Our sensor system consists of a textile underlay made of a fleece material which is placed underneath the actual floor covering. Within this layer a regular grid of microelectronics modules is arranged each of which is connected to a set of up to eight sensor plates. (see Fig. 1). A typical underlay contains 4 sensor modules and 32 sensor plates per square meter. This way, one sensor plate has the size of a typical human foot. However, higher or lower resolutions are possible.
When a person walks across the floor, sensor events are triggered for each sensor plate right beneath the person's feet. By means of a wireless transceiver on each microelectronics module, the sensor events are transmitted to a central receiver we call Smart Adapter Software on this receiver processes the succession of sensor events and reconstructs their location. Based on this inforFig 1. Textile underlay with microelectronic mation, the Smart Adapter can control modules and sensor plates and power lines wireless switches which can operate automade of conductive fleece. matic doors, alarm devices, lights, heating, traffic counters etc. It is also possible to connect the Smart Adapter to an already existing building control network. For simple applications, the wireless switches can react directly to the signals from the microelectronic modules such that a Smart-Adapter is not required. Fig. 2 summarizes a schematic of the general principle. Each sensor has a unique ID which is part of the messages transmitted to the Smart Adapter By means of this ID the position of each footstep can be determined. The Smart Adapter can be connected to a PC to display the sensor events or to do some higher level processing such as pattern recognition or behavior monitoring.
As can be seen from the schematic, it is also possible to send commands from the Smart Adapter to the modules. This can be used either to read out the status of the modules, to reconfigure their wireless characteristics or to request the temperature of the modules which is determined by the built-in temperature sensor on each module. By means of the latter, a temperature map of the entire floor can be obtained. 2.2
Each sensor plate (electrode) in the underlay is part of a so called open capacitor. Unlike conventional capacitors, which have two plates, the second plate of an open capacitor can be imagined to rest at infinity. This way, the sensor measures the capacity of the whole environment right above the sensor plate. Part of this environment is the floor covering, but also the air above the floor. Typical capacity values are around 100pF (base capacity). A person entering the volume element right above the sensor changes the dielectric properties of this capacitor. The capacity typically changes by 20-50pF. The exact value of the base capacity depends on the material of the floor covering but also on the environment's humidity and temperature. These effects are filtered out by a specific drift compensation in the capacity measurement algorithm. Likewise, the exact value of the capacity change depends on the vertical dimension of the floor covering, the size, thickness and material of the person's shoe soles and the size of the sensor plate itself. Not only humans but also large dogs or water containers lead to a sensor event.
Of course, metallic objects such as furniture, lead to a sensor event in those parts of the floor that are placed right beneath their base. However, these areas are not accessible to the users walk anyway. There is no influence of objects on sensors neighbouring the direct footprint of the object.
The micro controller on the sensor modules can determine the slight capacity change. This can be achieved by various methods. For instance by measuring the time for charging the capacitor once or by counting how many charge-discharge cycles can be achieved within a given time interval. It is possible to determine the capacity of multiple sensor plates independently.
To ensure a robust measurement over long time periods it is necessary to compensate for slow drifts of the base capacity due to changes of the environment's conditions. This drift compensation allows for another advantageous effect: when the person or object that triggered a sensor event, remains at the sensor location, the algorithm compensates for that initial capacity change and takes it for the new base capacity after some time. This way, additional capacity changes at the same sensor location (e.g. by a second person stepping on the sensor ) can be detected after compensating for the first sensor event. In practice, this effect occurs, for instance, when an additional piece of carpet is put on the sensor floor: initially, the sensors beneath the original floor may detect a capacity change due to the abrupt change of the environment's condition. After compensating for this change, the sensors automatically become fully functional again, even under the new carpet.
A very important issue is the right sensitivity tuning of the measurement algorithm. When the sensor is not sensitive enough, it will not work under thick floor covering or for certain shoe soles. Making it too sensitive, it may become erratic. We have tuned the sensors such that they will recognize human feet even when they are held a few centimeters above the floor covering which, in turn, may have a thickness of up to 5cm.
By calculating the direction of the capacity change it is possible to determine whether a person steps onto (ON-event) or leaves a sensor (OFF-event). This way, a person's current position can be inferred even when remaining static at a location.: as long as for a specific sensor only a positive capacity change has been recorded, “someone” is likely resting at that place. Of course, the robustness of this estimate can be improved by tracking the whole trajectory of that person. How this can be done is the topic of a later section.
However, the sensor signal itself is purely binary. A sensor plate is either occupied (ON-event) or empty (OFF-event).
In principle, any conductive material can be used as sensor electrode. As the overall thickness of the underlay is a critical factor, we work with conductive textiles that are attached as second layer to the basic non-conductive fleece (see Fig. 1). The capacitive measurement principle does not require a high conductivity for the sensor material. Therefore, a whole variety of conductive textile materials is possible: metalized fleece, fleece with a fraction of carbon fibers, textile with woven-in conductive wires, metal foil, etc. As the electrical parameters are not critical, the right selection of the sensor material depends more on factors such as price, reproducibility, mechanical robustness and longterm stability and the problem of how to connect the sensor material robustly to the microelectronics module.
However, we have come up with a specific geometry of the sensor layer, which allows for a simplified power supply of the modules: the narrow bands between the triangular sensor plates are power lines made from the same material as the sensors. As each module is connected to two power and two ground lines, a redundancy is given which allows for easy cutting and connecting of patches of the textile underlay. Only a simple switching AC/DC adapter connected to an arbitrary edge of the underlay is required for the installation. When structuring the sensor plates and power supply lines from the same material, the minimal allowed conductivity is governed by the requirements of the power supply for the modules and not by the capacitive measurement.
Based on these factors and on aging-tests with different materials and interconnect techniques, we ended up with two candidates which both seem to work for mass-production: a metalized fleece material and a synthetic textile with in-woven conductive wires.
To avoid the problem of having to transmit the sensor events over cables within the textile underlay we have decided for wireless data transport. Each microelectronic module is equipped with a radio transceiver which works at 868 MHz transmission frequency.
We have developed a specific message format which contains the ID of the sensor as its central element. Whenever a sensor detects an ON- or OFF-event, a message with the corresponding sensor ID is sent out by the module. The message can be received and interpreted by any wireless device that works on the same frequency. Two examples are the Smart Adapter and the SmartSwitch we will describe later. Also sensor modules that are currently not sending receive the sensor messages from the active modules. This allows for all types of network structures. It is possible, for instance, to program the sensor floor as multi-hop network to propagate sensor messages. However, currently we have implemented only the direct method in which the Smart Adapter or SmartSwitch directly receives the messages from each active sensor module. In principle, missing a data packet due to collisions in the transmission channel are possible. However, all applications of the sensor floor are based on evaluating whole populations of sensor events (e.g. trajectories) rather than on single signals. Therefore, missing some sensor events does not impair the overall function of the system. There is no interference with other radio technologies such as WiFi or DECT as the frequency bands do not overlap.
The message format we have developed does also allow for commands sent from the external receiver to the microelectronics modules. By means of a set of specific commands the behavior of the modules can be changed even though, after installation, the modules are physically unreachable beneath the floor covering. Among these commands are selective resets for the transceiver or the sensor IC. Further, it is possible to temporally deactivate specific sensors (e.g sensors beneath furniture) or to change the radio output power of the transceiver. The latter can be used to limit the range of the radio signals to the workspace where the sensor floor is installed in order to minimize radio interference with other wireless devices nearby. As the actual characteristics of the environment can only be measured after installation, it becomes handy having over-the-air access to the parameters of the transceivers.
The sensor events are collected by one central receiver in every room in which the sensor floor is installed. Also this Smart Adapter has a dedicated ID which is part of every message interchanged between the sensor modules and the Smart Adapter This way, the correct association between the modules and their corresponding Smart Adapter can be guaranteed even when multiple rooms are equipped with the sensor floor.
The function of the central receiver is to process the sensor events coming one by one from the sensor modules. The task is to analyses the time series and to reconstruct the movement trajectories of the people walking on the floor. Based on this information the movement history can be recorded and the intentions of people can be predicted as we will describe within the next chapter.
Currently, the Smart Adapter is connected to an ordinary PC via an RS232 serial connection. (via USB) Its only task is to collect the messages with the correct Smart Adapter ID and to send them to the PC where the further data processing takes place. Likewise, data to be sent to the modules or to the wireless switches are generated on the PC and transformed into wireless messages by the Smart Adapter. Future versions of the Smart Adapter will be embedded devices that fit in ordinary wall socket compartments and that do all the sensor data processing on board. For the communication with conventional home automation networks and devices, the SmartAdapter will be equipped with the appropriate interfaces (LAN, WLAN, LON, etc.). To control the effectors of the applications such as doors, alarms and lights, we have developed radio modules that look very similar to the sensor modules. However, instead of the capacitive measurement algorithm, these wireless switches contain a set of electronic relays which can switch arbitrary devices. They are also equipped with binary input channels that can detect the state of the devices. This way it is possible for the Smart Adapter to open a door for an approaching person and to simultaneously check the state of a light barrier in the doorway to detect obstacles which may not have activated the capacitive sensors.
As pointed out earlier, the wireless switches can be controlled directly by the sensor modules for simple applications which do not require a Smart Adapter Fig. 3 shows an example in which the sensor floor directly controls a light panel on the wall.
The basis for all applications is an initial mapping of the sensors' IDs to their locaFig. 3. Demonstrator with different floor tion within the room. This mapping must icnogveorningthse(cflaoropretw,igrlealsess,sllyamcoinnatrteo)l.s WLEalDks- be done only once right after the installaon the wall. tion and can be accomplished in various ways. If the geometry of the underlay has not been changed (e.g. due to cutting during installation), only the global position of the underlay in the room has to be announced to the Smart Adapter because the relative position of the sensors with respect to the underlay is already known from the production process. In the other extreme, when the original mapping is totally lost, the room can be mapped by walking a path which activates the sensors in a defined sequence (e.g. a zigzag-course). In any case, the Smart Adapter stores the association between the sensors' IDs and their metric position in an internal table. The various applications for the sensor floor put different demands on the complexity of the data processing. For a very simple application, such as a presence-detector, it is enough to monitor whether any sensor messages come from a defined set of sensors on the floor. A special version of this is shown in Fig. 3: every light on the wall listens to its dedicated sensor. The fine spatial resolution of the sensors system (32 sensor plates per square meter) allows for an exact definition of the floor areas in which the activity is measured.
Also, activity monitoring can be implemented easily: a histogram map which counts
the on signals for every sensor can be directly interpreted as a measure how much
walking activity occurred at which location. A protocol of these figures over a long
time period allows for a detection of abnormal variations in peoples' activity.
Somewhat more advanced is the tracking of individual persons. We have implemented
methods from the theory of nonlinear dynamical systems (see e.g. [
e a s t −xit 2 b2
(0)
Herein, a is the so called time scale and b is a spatial range. Evaluating this equation
by integrating dx/dt over time leads to the effect that the position estimate x is
attracted with a strengths 1/a by those sensor inputs s which occur within the range b around
the current position estimate. The sensor signals play the role of attractors in this
dynamics (see also [
A similar dynamics is chosen for the estimates of the person's movement direction Φi and velocity vi. Choosing the parameters appropriately, the position estimate can even be interpreted in a predictive way: assuming that the velocity and movement direction of a walking person will usually not change abruptly but smoothly, the probable intention of a person can be inferred from the sensor signals: if a person walks towards the entry of a room, for instance, the light can be switched on in advance. By means of the dynamical systems approach, the problem with crossing movement trajectories while tracking two persons can be minimized as well. As the dynamics represents the movement state of a person which is only modulated by the sensor input, this state is preserved even during a period in which the sensor information is missing or can not be disambiguated. This means, for instance, that two people the movement paths' of which intersect and the sensor signals of which can not be separated anymore, are still represented as individual persons and can be tracked separately again later on.
This approach also eases the counting of people in a room. However, an appropriate method must be chosen for deciding when to assign a new position estimate (i.e. a new dynamics (0)) to a cluster of sensor signals and when to remove a void estimate. This occurs, for instance, when a person enters or leaves a room. Regarding this problem, we are still in the research phase. However, several assumptions can be exploited for the decision making process, e.g. the known location of the room's entrances. Finally, some applications require the identification of static sensor patterns on the floor. This is needed, for instance, to detect the fall of a person. Here, we plan to implement pattern recognition algorithms known from image processing. Except for the simple applications for which the effectors can be directly controlled by the sensor modules, the data processing takes place within the Smart Adapter. However, the configuration of the application is heavily dependent on the actual ground plan of the installation site. To open a door, for instance, the Smart Adapter needs to know where the doors are and from where people typically approach this door. Therefore, it is necessary to investigate the installation site and to plan the software correspondingly. For this project planning, it is also necessary to know the existing infrastructure such as already existing effectors or building automation systems, to which the sensor floor should be interfaced. To deal with this task, we have started to build an integrated simulation environment which can incorporate ground plans and typical movement trajectories of the inhabitants.
As a future vision, we plan to build up self-learning algorithms which extract the knowledge about typical movement trajectories and desired effector control from a training phase after installation. A typical scenario for such a system would be a software in the Smart Adapter, which permanently collects sensor events and associates them with actions such as people opening doors or switching on or off lights. After the training phase, the Smart Adapter would just take over and control the effectors in the same way whenever a sequence of sensor events is discovered that resembles one from the training phase. Ideally, such a system would not require any project planning at all and even the initial mapping of the sensors to their location in the room would be obsolete.
There already exist pressure sensors which can detect the presence of people (see e.g.
[
For the application scenarios described in the next section, our capacitive proximity sensors have several advantages compared to the pressure-based sensors. First of all, a pressure sensor requires a soft floor covering (e.g. carpet or linoleum). Otherwise a high spatial resolution can not be achieved. Our aim is not to interfere with the individual design of the installation site. A proximity sensor can be placed under hard floor coverings a s well. Second, our sensor system does not contain any mechanical parts. Pressure sensor which are based on a material that ensures a certain distance between the sensor plates wears out after some time. Finally, our system provides the unique feature of self-testing: the state and integrity of the sensor system can be requested at any time by means of a wireless request from the central receiver.
As mentioned in the introduction, one of the intrinsic characteristics of our sensor floor is the fact that the technology remains completely invisible to the user. It is only the function that becomes apparent. From a user's perspective, the sensor floor's functions can be classified into several categories which we will describe in the following.
Probably the most obvious application of a sensor floor is the control of heating, air condition and illumination. Unlike conventional infrared sensors which just detect general movement, the sensor floor can count the number of persons in a room and is able to track this number even when people remain static. It is even possible to dim the illumination at empty parts of a large room or of an openplan office.
Using a piece of sensor floor to control an automatic door, can also contribute to energy saving: opening the door only for a person whose movement trajectory Fig. 4. Equipped with the sensitive floor, an indicates the intention to pass the door automatic door opens only when people intend prevents its unintended opening whento enter ever someone enters the range of the movement detector without intending to pass the doorway (see Fig. 4).
In addition to opening automatic doors, the sensor floor can support more comfort functions within public buildings. Analyzing the movement trajectories it is possible to order an elevator already when someone walks towards it.
Initializing the movement tracking to a person at the check in, it is possible to give individualized direction information in hotels, hospitals or other large public buildings. In private homes, comfort facilities such as the hot water pump can be activated already when the floor detects the inhabitants' first activity in the morning. Likewise, home appliances can be switched off when the last person has left the house. A sensor floor mat in front of the bed can decide whether a person leaves or enters the bed and control the room lights accordingly (see Fig. 5 right panel).
By analyzing trajectories, the sensor floor can act as an intelligent burglar alarm: in contrast to trajectories starting at the room's door, footsteps starting at a window are associated with intruders.
Implemented in a sally port (security entrance), the floor can detect people entering without using the identification terminal. Likewise, the floor can detect when two persons enter with only one carrying an identification tag.
At night, security personell can remotely monitor the activity of rooms in which the sensor floor is installed.
In the case of an emergency, the sensor floor can support rescue personell by providing statistics about how many people have been in which part of a building. when the fire had started.
In areas where people live or work together with robots or other mobile machines, the
floor can detect the danger of potential collisions between people and machines. When
people enter the workspace of robots, for instance, the machines can slow down their
movements. Likewise, transport carts can be warned and slowed down at intersections
when there is the danger of a collision with people even when the vision is impaired
by walls or furniture (see [
As the demographic structure of the society changes towards a larger percentage of elderly people, health care becomes a very important field of application for the sensor floor. In particular for people who want to avoid moving to a nursery home but want to live in their own apartment as long as possible, the sensor floor can offer various assistive functions. A simple example is a mat of sensor floor besides the bed which switches the room lights off when the user steps into bed and which switches the lights on when the person steps out of bed. Already with such a simple system, many accidents of people who forget to activate the illumination at night can be prevented (see Fig. 5 right panel). Analyzing the inhabitant's movement trajectories, the floor can detect when a person falls and cannot get up anymore (Fig. 5, left panel). Likewise, the floor can detect when the inhabitant has not left the house but remains inactive for a long time. In both cases, a call for help can be triggered automatically over the telephone. In this field of application, the floor has many advantages compared to conventional systems. A wearable alarm device, for instance, does not work when the user is unable to activate it after a fall. Further, many people put the devices away when showering such that accidents in the bathroom remain undetected. Observing the condition of elderly people by means of surveillance cameras usually requires a human to analyze and interpret the video images which impairs peoples' privacy.
An interesting application of the sensor floor in the medical field is the long term
activity monitoring of patients with mental diseases. From rapid changes of the activity
level above or below the average, phase transitions of certain diseases such as
depression or schizophrenia can be predicted and treated in advance.
It is also possible to derive information about the general state of health from
analysing a person's gait pattern [
A fascinating application is the coupling of peoples' location with acoustic signals. We have developed a MIDI interface which can generate different sounds depending on the coordinate of the sensor signals. This way it is possible to provide a blind person with an acoustic map of the environment. The access of interesting or dangerous locations can be indicated by specific sounds. Even a potential collision with dynamic obstacles, such as other people, for instance, can be indicated to the blind person by means of special warning tones. 3.4
By analyzing peoples' movement behavior in public buildings, the sensor floor can provide important information for marketing purposes building planners and facility management. Visitor streams can be analyzed without interfering with peoples privacy as the floor, by design, cannot detect the identity of people.
Some of the first customers of our system come from the domain of trade fairs and entertainment. Here, the sensor floor can equip an installation with interactivity: illumination and multimedia is controlled by the visitors as they move through the site.
In this paper we have presented an innovative sensor system which is based on capacitive proximity sensors arranged in a textile underlay underneath conventional floor coverings. The sensor signals triggered by people walking across the floor are transmitted wirelessly to a central receiver. There, the movement trajectories are reconstructed by means of a nonlinear dynamical systems approach. Based on this information many different applications in the domains of AAL, health care, comfort, energy saving, security, marketing and entertainment are possible.
An important feature of the system is its seamless and invisible integration in the environment without disturbing the design or the privacy of peoples' area of life. Only few applications have been implemented so far. Further research topics are in particular the processing of the sensor data to extract high-level features such as static patterns after a fall and the incorporation of algorithms which can autonomously extract interesting features from the long-term characteristics of peoples' movement patterns.