We propose a multi-pedestrian tracking system based on MEMS based IMUs as a novel tool for human behavior analysis. With asynchronized multiple IMUs, our system can track IMU-attached pedestrians in synchronization at a high frame rate in the large environment, compared with vision based approaches. The output data is similar to standard PDR systems as follows: the time-series position, velocity, and heading of the pedestrians in the 3D space. To realize our system, we propose a simple but e ective calibration technique for synchronizing the timelines of the asynchronized IMUs. With our system, users can analyze the detailed motion behaviors of the people who participate in a group work or a collective activity, quantitatively. By combining with other sensors such as an eye tracker, our system can further provide more comprehensive data in the experiments.
Various demands on indoor positioning systems also has been rapidly
increasing for location-based services, such as navigation to shops, and advertisements.
From a technical point of view, the indoor positioning systems can be classi ed
into three categories: radio wave based systems, vision based ones, and inertial
sensor based ones. The use of radio frequency identi cation (RFID) [
In this paper, we propose a multi-pedestrian tracking system based on MEMS
based IMUs. Our goal is to track multiple IMU-attached pedestrians in
synchronization at high frame rate in the large environment, compared with vision based
approaches. Especially, we tackle the case of using asynchronized IMUs because
the global time based synchronization for multiple IMUs is not always usable, or
not accurate. In our prototype system, we use the NGIMU1 as an IMU device.
The use of our system is designed as follows. First, we x all of the IMUs on the
board, and rotate them for the process of time-synchronization. Next, we attach
the IMU onto each pedestrian, and collect data during the experiments for
human motion analysis. After the experiments, we synchronize all of the data, and
nally output all of the tracking results. For the behavior or motion analysis
for each pedestrian, we can use any PDR system, such as [
In summary, the contribution of our paper is threefold as follows. { A novel multiple pedestrian tracking systems is proposed. { A calibration process for o ine time-synchronization of multi-IMUs is proposed.
{ A potential usage is introduced. 2
Our system is designed for tracking multiple IMU-attached pedestrians in the large 3D environment. First, we attach asynchronized IMUs onto pedestrians to capture the data during the movement, and then process the data o ine. The de nition of asynchronized IMUs is that they have their own timelines 1 http://x-io.co.uk/ngimu/ Capture Processing timer
IMUs recorded time points
recorded sensor data data synchronizer synchronized sensor data
single PDR synchronized tracking results
data viewer individually such that the timing of booting an IMU corresponds to the origin of the timeline. Since it is not easy to simultaneously boot all of the IMUs, for instance 10 IMUs, the time synchronization technique is required for our system. As illustrated in Fig. 1, our system is based on two main steps: the data capture during the experiment for human behavior analysis, and the data processing after the experiment. The modules in the processes consist of four parts: the timer, the data synchronizer, the single PDR, and the data viewer. In the rest of this section, we explain the detail of each module, and its usage. 2.1
The timer is a simple tool to save time points for all the processes. During the experiment, some important time points should be manually recorded by the users: when the rst IMU is booted, when the last IMU is booted, when the calibration data capture is started and ended, etc. This tool is normally installed on a laptop so that the time points are recorded in a global timeline given by a laptop. 2.2
We propose a software based time synchronization technique to use asynchronized IMUs for multiple pedestrian tracking. As illustrated in Fig. 2, the global
Angular velocity 0 Angular velocity IMU2 timeline
IMU1 timeline
Global timeline time de ned in the timer is the main timeline in our system. In this section, we rst explain how to use the timer for the data synchronizer, and then explain how to synchronize the data from di erent IMUs.
Usage and timeline Here is the example of using two IMUs for explanation simplicity. If multiple IMUs are used, IMU2 corresponds to the lastly-booted IMU, and IMU1 corresponds to the rstly-booted IMUs. First, the users record the time t1 with the timer, as the time point before booting all the IMUs. Then, they boot IMU1 at time t2, and do the same for IMU2 at time t3. These are not recorded. We assume that each IMU starts recording the data to the internal memory card immediately right after it is booted. For an asychronized IMU, the booting time corresponds to the origin of each IMU timeline. Therefore, the time points of each origin in each IMU timeline are di erent in the global timeline. The users record time t4 with the timer, as the time point after all IMUs are nally booted. They start a speci c motion for calibrating the IMU timelines at time t5, and nish the motion at time t6. It should be noted that we do not record t2 and t3 in the timer because it is not easy to manually save the accurate time point. This means that we do not know the exact time of t2 and t3 in the processing. t1, t4, t5 and t6 are roughly necessary for the time synchronization.
Next, we explain the main idea of how to merge the IMU timelines onto the global timeline. As discussed before, the exact value of t3, when IMU2 is booted, is not known. Therefore, in the IMU2 timeline, it is not possible to nd the exact correspondence of time points which are recorded in the global time. To solve this problem, we use t4, which is recorded with the timer after IMU2 is booted. Before recording a next time point tx after t4, the IMUs are stationary for a time ts, which should be larger than t4 t3. This stationary process is useful to segment the data. In the IMU2 timeline, the IMUs are stationary between tx t3 ts and tx t3 of the global timeline. We assume that t4 t3 is small (a) Step 1 (b) Step 2 (c) Step 3 (d) Step 4 (e) Back to step 1 enough (less than 10s), and use ts = 10s. For most researches, we do not need the exact correspondence of tx in each timeline, and can use tx t4 as the approximation instead. It is easy to nd that the IMUs are also stationary at this time (tx t4 2 [tx t3 ts; tx t3]). Therefore, using this approximation will not introduce any error on positioning.
User interaction for synchronization Hereafter, we match the IMU1 timeline to the IMU2 timeline for the time synchronization. When using more than three IMUs, we sequentially match the IMU1 timeline to others. To use the time synchronizer, the user interaction is required in the step of the data capture.
In this process, we propose to use a board to x the IMUs on it, and move the board to generate some unique data changes in the angular velocity space. After all the IMUs are booted, we x the IMUs on the board such that their z-axis on the IMU frame is perpendicular to the board. Then, the IMUs are stationary for ts = 10s for the data segmentation. After this, we rotate the calibration board around z-axis on the ground or a desk, as illustrated in Fig. 3. This movement should be designed for matching two sequential data accurately. In our empirical experiments, we found that the su cient amplitude of the rotation was around 30 for more than 10 rounds. At the end, the IMUs are again stationary for ts = 10s. To compute the time o set between IMU1 timeline and IMU2 timeline, we match the angular velocity data on z-axis.
Computation of time o set The time o set minimizing the least absolute deviation (LAD): t = t3
t2 is computed by t=tmin tmax argmin X jIM U 1:datat+ t t
IM U 2:datatj t 2 (0; t4 tmin = t5 (4)
Algorithm 1: Matching the IMU1 timeline to the IMU2 timeline for IM U 1i in IM U 1:
IM U 1i: t = N one IM U 1i:LAD = 0:0 for t in range(0, t4 t1):
LAD = 0:0 for t in range(t5 t4, t6 t4):
LAD = LAD + jIM U 1i:datat+ t IM U 2:datatj if (IM U 1i: t == N one) or (IM U 1i:LAD > LAD):
IM U 1i: t = t
IM U 1i:LAD = LAD Angular velocity 0
Global time IMU 1
IMU 2 where IM U 1:data is the angular velocity of IMU1, IM U 2:data is that of IMU2. As illustrated in Algorithm 1, the optimal t is found by a slide window based approach. Finally, as illustrated in Fig. 4, the time o set t can be computed so that all the IMU timelines are matched to the global timeline. Theoretically, the error on computed t is less than 1:0=rs, where rs is the sampling rate. We can use any single PDR system such as foot-mounted or chest-mounted one to generate the tracking result for each pedestrian. The mount positions can be selected according to the issues of human motion analysis. With the synchronized sensor data, it is easy to merge all of the results in the one timeline. 3
We show the behavior analysis of two persons: a person who guides a person, and a person who is guided, namely the guide and the follower. We attach the 2.5
IMU on each right foot, and use a standard foot-mounted PDR system based on dead reckoning and map matching for generating each tracking result.
With the synchronized tracking results, it is possible to analyze the relationship between the walking status of two people: the guide and the follower. For example, as illustrated in Fig. 5, the relationship between the step length of the guide and that of the follower along the traveled time can be generated. Generally, the follower tries to walk as the guide does. With vision based approaches, it is not possible to acquire the step-level human motion in the large environment. By using our synchronization technique, we can provide the tracking results of multiple pedestrians. From a psychological point of view, it would be interesting to analyze the degree of the comfort for the follower. 4
In this paper, we proposed a cost-e ective multi-pedestrian tracking system based on MEMS IMUs. Our system supports the researchers who are working on tracking multiple pedestrians for human motion analysis. With our calibration process, the synchronization error on time was less than 50ms, which can satisfy most of research purposes. Our data viewer can be a tool such that the researchers can intuitively analyze the position and heading of the pedestrians in the experiments.
In our future work, we apply this system to the human motion analysis in
crowd [