Our research group is developing a marker-based optical motion capture system using smartphones in a mobile setting. The proposed system can be realized with low cost, high accuracy of measurement, and can be easily deployed. This paper describes a 3D tracking technique using smartphones, which is one of the key parts of the proposed motion capture system. The system requires multiple smartphones, an LED light source, and speakers. The 3D coordinates of the cameras are obtained by acoustic localization, and the 3D coordinates of the markers are obtained by the principle of triangulation from the obtained images and the 3D coordinates of the smartphones. 3D tracking experiments using two smartphones in xed settings con rmed that the proposed system could achieve millimeter-level accuracy. Its 3D tracking performance in a handheld setting was also tested.
A motion capture system (MCS) is used in scenarios such as sports science, medical research, the creation of CG models in games, and movies. Among them, a marker-based optical MCS is the most popular MCS because it can measure with high accuracy. However, the commercialized optical MCS is extremely expensive at a cost ranging from tens to hundreds of thousands of US dollars. In addition, it is necessary to accurately measure the positional relationship between multiple cameras, requiring high deployment costs. As a result, such systems are usually used in a xed nonmobile setting. In contrast, smartphones have become ubiquitous in today's world. Smartphones contain different types of sensors such as cameras, speakers, microphones, and inertial sensors, and they can be used for various applications. Our research group is developing an MCS using smartphones, thereby making the system mobile. The proposed system can be realized at a low cost because it employs users' smartphones, an LED light source, and three loudspeakers. Moreover, the proposed system can measure the positions of markers with high accuracy and can be easily deployed without manually measuring the positional relationship between cameras. This paper evaluated the performance of one of the key technologies of the proposed system, 3D tracking, and con rmed that it could achieve millimeter-level accuracy in estimating the 3D positions of moving markers. The system was also tested in a handheld setting to verify its 3D tracking performance. 2
There are various types of MCSs; those using cameras are called optical MCSs
and comprise two major types. One is marker-based (e.g., [
A marker-based optical type MCS was investigated to achieve higher accuracy in a mobile setting. The proposed system mainly performs three processes. The rst process is to perform localization of smartphones and obtain their 3D positional relationship. The second process is to conduct the frame interpolation between smartphones so that captured images by different smartphone cameras are time-synchronized. The third process determines the 3D positions of markers to be attached to a person's body and captured by each smartphone using triangulation. This paper describes the current implementation status of the system using two smartphones. 3.2
For the localization of the smartphones, the time-of-arrival (ToA) trilateration
method [
Although a reference point of time synchronization for ToA trilateration is accurately identi ed by individual smartphone cameras as discussed in Section 3.2, their shutter timings are different from each other. In Fig. 1, differences between a signal emission time of an LED light and shutter timings of smartphones 1 and 2 are obtained as t1d and t2d, respectively. Therefore, a moving target is captured at different times by different smartphones, which produces localization errors. To solve the problem, a frame interpolation method is applied to obtain the target image at the same timing by different smartphones,
As shown in Fig. 1, a target is captured at the k-th frame by the smartphone
1 and k′ -th frame by the smartphone 2, and the difference between their shutter
timings is t2d t1d. Suppose that the positions of the target are given as (u1;k; v1;k)
and (u2;k′ ; v2;k′ ) in the images captured by the smartphones 1 and 2, respectively.
By conducting linear interpolation using the k′ -th and (k′ + 1)-th frames of the
smartphone 2, an interpolated target position (ui12;k+1; v1i2;k+1) corresponding the
(k + 1)-th frame of the smartphone 1 is estimated as
1
ui12;k+1 = (u2;k′ +1 u2;k′ )f2C ( f C (t2d t1d))
1
1
v1i2;k+1 = (v2;k′ +1 v2;k′ )f2C ( f C (t2d t1d));
1
where f1C and f2C are camera frame rates of the smartphones 1 and 2,
respectively.
When internal and external parameters of cameras are known, the 3D
coordinates of markers on the images captured by cameras are determined. The
internal parameters are those related to focal length, image center, and
distortion, and the external parameters are those related to rotation and translation.
The internal parameter matrices A1 and A2 of two smartphone cameras are
assumed to be known using the camera calibration method [
2pi;11 pi;12 pi;13 pi;143 Pi = 4pi;21 pi;22 pi;23 pi;245 :
pi;31 pi;32 pi;33 pi;34
Qw is found by solving b = BQw, where B and b are given as Eqs. (4). The least squares solution of the target point Q^w is given by Eq. (5). B+ is the pseudo-inverse of B, and this method can be applied when the number of cameras is three or more.
2p1;31u1 B = 664pp21;;3311uv12 p2;31v2 p1;11 p1;32u1 p1;21 p1;32v1 p2;11 p2;32u2 p2;21 p2;32v2 p1;12 p1;33u1 p1;22 p1;33v1 p2;12 p2;33u2 p2;22 p2;33v2 p1;133
2p1;14 p1;23 77 ; b = 664pp21;;1244 p2;135 p2;23 p2;24 p1;34u13 p1;34v1 7 p2;34u257 p2;34v2 ^
Qw = B+b; B+ = (BT B) 1BT Ideally, once the transmitter and smartphones are synchronized, the proposed method can conduct ToA trilateration for tracking smartphone positions by (3) (4) (5) using only acoustic signals as discussed in Section 3.2. However, this is not possible in practice. Fig. 2 shows the phase values of optical signals acquired by the left smartphone in 30-s measurements; it was con rmed that the phase value changed during the measurements. The working frame rate of the smartphone camera of 60.001 fps, despite having set the rate to exactly 60.0 fps, affected the phase value. As a result, the standard deviations of the smartphone localization in the measurements deteriorated, as shown in Table 1 (\without calibration").
Fig. 2 shows that the phase value changes at a constant rate. Thus, it is possible to conduct calibration by linear regression. In Table 1, \with calibration" means using a smartphone with the calibration by linear regression, and \FG synchronized" means that the optical signal generated by a function generator was accurately set to one-third of the smartphone camera's frame rate (60.001 fps). The table shows that the localization results of the smartphone calibrated by linear regression were improved to almost the same level as the localization results of the FG synchronized smartphone and that the calibration by linear regression proved to be effective. In our investigation, because actual frame rates set to 60 fps differed between smartphones, their frame rates should be measured and the linear regression for each smartphone should be conducted.
X Y Z with calibration 2.79 3.511 2.27 without calibration 14.63 26.19 35.73
FG synchronized 2.77 2.87 3.46 The experimental setting is shown in Fig. 3. Two smartphones (iPhone 6s Plus) were used for tracking two color markers attached to a moving square bar under a uorescent lamp in a room. A transmitter was placed at a line-of-sight position from the smartphones so they could capture transmitted optical and acoustic signals. Then, three speakers (FT200D, Fostex) mounted on the transmitter formed an equilateral triangle with a side length (baseline) of 300 mm, and the light source consisting of 56 white LEDs (OSW54L5111P, OptoSupply) was placed at the center of the triangle. The smartphone cameras had 1920 x 1080 pixels and were operated at 60 fps. Then, 20 Hz sinusoidal signals whose frequency was one-third of the camera frame rate were generated by a function generator (WF1948, NF Corporation) and emitted from the LED light. Sync patterns transmitted from the three speakers were composed of a pair of sinusoidal waves whose frequencies were 12.75 and 13.25 kHz, 13.75 and 14.25 kHz, and 14.75 and 15.25 kHz, respectively. As they were emitted every 0.2 seconds so the position update rate of the smartphones was 5 Hz. The linear regression was conducted to compensate the clock difference between the smartphone as discussed in Section 4.1.
By receiving signals from the transmitter, the 3D position of each smartphone was calculated. Then, the position, pose, and captured image data were transferred from the smartphone to a PC via WiFi. The application software running on the smartphones was implemented using Swift 4. The 3D position estimation of markers was conducted using OpenCV 4.0 on the PC, and the measurements were carried out 10 times.
Two smartphone settings were tested in the experiments: Experiment1 xed on a tripod as shown in Fig. 3, and Experiment2 held by a user.
The baseline between the two tripods was set to 0.39 m in Experiment 1 and users holding a smartphone stood at the same positions of the tripods in Experiment 2. The distance between the markers and smartphones ranged from 0.94 m to 1.04 m. Tracking results obtained by the proposed system were compared with those by a high-performance and commercially off-the-shelf motion capture system (MAC3D System, Motion Analysis Corporation) composed of 6 cameras (600 fps, 2048 x 1088 pixels) as the reference system. 3D tracking errors of the proposed system were given as differences of estimated marker 3D positions from the reference system. 4.3
Fig. 4, Fig. 5 and Table 2 show the tracking results. In Fig. 4 (a) and (b), the
bar was translated quickly (captured frames: 27) and slowly (captured frames:
82) in the horizontal direction. As shown in Table 2, root means square
errors (RMSEs) of the marker tracking were 7.15 mm (marker1, quick), 5.44 mm
(marker2, quick), 2.20 mm (marker1, quick) and 1.93 mm (marker2, slow),
respectively, without applying the frame interpolation method. The improvement
of the tracking performance ranged from 0.01 to 1.99 mm with this method. Fig.
5 (a) and (b) show the results when the bar was translated quickly (captured
frames: 21) and slowly (captured frames: 75) in the vertical direction. As shown
in Table 2, RMSEs were 1.89 mm (marker1, quick), 2.33 mm (marker2, quick),
2.77 mm (marker1, quick) and 1.34 mm (marker2, slow), respectively, without
applying the frame interpolation method. The improvement of the tracking
performance ranged from 0.10 to 0.34 mm with the frame interpolation. Fig. 6 shows
trajectories of the markers translated quickly and vertically (Fig. 5(a)). From
the gure, the estimated trajectories are more unstable in the z-axis (depth)
direction than in the x and y-axis directions. According to the theory of dilution
of precision (DOP) [
This paper proposed a 3D tracking system using smartphones for a markerbased optical MCS in a mobile setting. The proposed system did not request the positions of the cameras to be set manually and could estimate them automatically by ToA trilateration using optical and acoustic signals. The experimental results showed that the proposed system could achieve 3D position estimation of markers with millimeter-level accuracy in xed settings. A number of issues remain to be investigated in our future work. The next step is to extend the current system with more than two smartphones toward more accurate and precise motion-capture systems. 3 2 1 0 1 2 5 marker 1 x 5numbe1r0 of fra1m5es marker 2 om ]m tfr [m2 n t e in1 cem gpo0 la i
n p t
r isD ts-1
a om ]m tfr [m2 n t e in1 cem gpo0 la i
n p t
r isD tsa-1 20 20 . proposed system (w/o interpolation) . reference system 20