With the increasing demand for indoor positioning, VIO (visual inertial odometer) are receiving more and more attention due to the universality and convenience of the camera. We found that the visual observation of VIO is more susceptible to the environment, and the error of observation results in the final positioning error. For this reason, we specifically analyzed the visual error source for different scenarios and use the short-time reliability of PDR（Pedestrian Dead Reckoning). PDR is used to assist in the monitoring of integrity of visual observations.
positioning system that based on PDR only require the data of IMU. The system uses the physiological characteristics of pedestrian walking to estimate the pedestrian's trajectory by detecting the step size and stride frequency of pedestrian walking. The source of information for the VIO system is the binocular camera and the IMU, while the PDR has only one source of information is IMU. When using the VIO system for indoor pedestrian positioning experiments, they unable to obtain accurate visual observations in special environments (such as low light and light imbalance) which resulting in large positioning errors. We analyzed the visual measurement part and positioning errors of VIO. And we proposed autonomous integrity monitoring of a vision based pedestrian dead reckoning system. Which the short-term reliability of PDR (Pedestrian Dead Reckoning)[10] is used to assist in the visual integrity detection of VIO systems. 2
Visual inertial odometers are generally divided into fore-end and back-end. The fore-end mainly deals with sensor’s observations, such as feature extraction and integration of images with IMU data. The back-end mainly processes the residuals caused by the observations through a filter or an optimization scheme, and obtains an optimal position state. The structural block diagram of the system is shown in Fig.1. The k-th state vector Xk in visual inertial odometer can be defined as Xk including the
T state XIMUk of the IMU and a camera pose (attitude dqCTk and position G pTCk ). The measurement model of the feature point is expressed by the following equation Xk = êéëXTIMUk δθTCk
G !TCk úûùT ， zi = C1 é C X ù p ê C i ú + ni ,i Î f ,C Î{C1,C2}
Zi ë Yi û
(
C pi = êê CYi ú = C( Gi q)( G pi - G pC ) where G pi is the 3D feature position in the global frame, G pC is the camera in the global frame and C( CGi q) is the rotation matrix between the camera frame and the global frame. Once the estimate of the feature position is obtained, we can compute the measurement residual: ì ¶zi × ¶ C1 pi + ¶¶Cz2pii × ¶¶CX2 Cpi ri = zi - z"i # HC X + Hi Gp!i + ni ， where íïïHC = ¶ C1 pi ¶XC1 1 ï ¶zi × ¶ C1 pi + ¶¶Cz2pii × ¶¶CG2ppii ïîHi = ¶ C1 pi ¶ G pi where HC and Hi are the Jacobians of the measurement zi with respect to the state and the position estimate of feature. With all the sets of measurement equations formed by the feature points, we can get the optimal solution by minimizing the error and get the optimal position estimate.
é d Xˆ úù =(HT H)-1HTr ® Xˆ = X +d Xˆ， G pˆ i = G pi +d G pˆ i
êëêd G pˆ i úû
(
We found that the system has a large error in the positioning results obtained under special environments during the experiment. So we analyzed the error source and divided it into the following four error situations.
Insufficient features. Commonly used feature extraction algorithms include SIFT[11], SUFT[12], FAST[13], ORB[14] and other feature extraction algorithms. And those feature extraction algorithm is often used in processing of VIO projects. In fact, it can be derived from the definition that it is desirable to find a point with strong contrast with surrounding pixels as a feature point. The contrast of point P can be expressed as
V( x, y) = å I ( x +!x, y +!y) - I ( x, y) ， V( x, y) " åéëIx2 ×!x2 + I y2 ×!y2 + 2I xI y ×!x!yùû (
environments and we use the Lambertian model as the lighting model.
I ( x, y) = r ( x, y) × h( x, y)T × S Uneven distribution of features. It can be seen in the observation equation of the image that the presence of noise causes positional errors in the feature points in the image. The position error of the feature points will affect the state estimation of the camera when calculating the re-projection error. We describe the quality of the position estimate based on the camera state's Jacobian matrix of feature points Hi . In our model, the measurement error is zero-average, so the positioning error is also zero-average, so we can obtain the expected value and covariance of the error in the position calculation.
E (DX) = E ( Xˆ - X) = 0， Cov[DX] =s 2 (HiTHi ) -1 b2 - 4ac The amount of change in the position error in the x，y and z directions is represented bys x2,s y2,s z2 respectively. Use Hii to represent the first element on the diagonal in the diagonal matrix (HiT Hi )-1 . Then can be expressed as Where I( x, y ) is the image gray value, r ( x, y) is the object reflectivity, h( x, y) is the surface normal vector and S is the light intensity. We found that feature tracking is easy to lose leads to inaccurate positioning during light changing. The assumption of the optical flow method assumes that the gray level is unchanged. Substitute the lighting formula: ¶I dx + ¶x dt ¶I dy ¶y dt
éµ ù é ¶hT × S + ¶S = - ¶I ® éëIx I y ùû êën ûú = -r × ê ¶t ë ¶t ¶t
ù × hT ú û Where Ix and I y are the gradient values of the feature points in the x and y directions, respectively, µ and n are the velocity of the motion in the x and y directions representing the feature points. When the light intensity changes, the residual of the feature points caused by the illumination change can be expressed as d ri = C
1Zi êëé CCXYii -- ((CCYXi i++nµt)t)úûù = CtZi êëéIIxy úûù × êëér ¶¶St × hT úùû
SD(DX) = s x2 +s y2 +s z2 = H11 + H22 + H33
(
êë Ci Z j +!z ûú é Ci X j +!xù êê CiYj +!y úú = C( CGi q)( G p f j +!G p f j - G pCi )® ri = CZi (C!Zzi +!z) éëê CCXYii úùû - C 1 é!x ù
ú (
The error bound of PDR. Although PDR has a problem of cumulative error, the error
in a short time is very small. Now assume that there are two sampling points O1,O2 , the
sampling interval is Dt and the velocity of time O1 is v , the displacement is s , and the
state covariance matrix is P1 = êé p11 p12 ùú .Accelerometer observations can cause
inacë p21 p22 û
curate deviations due to shocks generated during motion. We first analyze the
onedimensional motion, the acceleration at time O1 is fmea = ftrue + d f .An estimated value
of the state quantity at O2 can be obtained from the state quantity at O1 . The deviation
caused by d f is:
(
PDR-assisted visual integrity monitoring. Hypothesis deviation obeys Gaussian distribution e ! N (0, å) . Now e is a three-dimensional vector. In order to facilitate the calculation, the inner product of the computation vector is transformed into a scalar .
ae -1 öT ae - ae r = eT å e = ç å 2 e ÷ ç å 12 e ÷ö, whereç å 12 e ÷ö ! N (0,I) è ø è ø è ø é êd f × Dt çae 1 Dt - p12 + Dt × p22 ö÷úù éd sù ê è 2 p22 + R øú êëd vûú = ëêêê d f × Dt èçae1 - p22p2+2 R øö÷ ûúúú
It can be thought of as the sum of the squares of two independent random variables subject to the standard normal distribution, which obeys the chi-square distribution of three degrees of freedom. The probability distribution (cumulative distribution function) isa = F ( x) , and given a a , we can determine an interval éë0, F -1 (a )ùû , and F -1 (a ) is the threshold we are looking for to determine the visual integrity.
Insufficient features. As shown in Fig.2(a), We changed the feature point extraction threshold for the same set of data, and the control variable is the number of features per frame. Moreover, the number of feature points is 0 at the 1560-th frame, because the white wall is encountered and the feature points cannot be extracted. We draw the corresponding positioning trajectory. As shown in Fig.2(b). When the feature points are scarce, the camera's ability to correct the IMU is not strong, the path is not serrated enough, and the trajectory also shows significant deviations between the x-axis and the y-axis. Lighting causes the failure of feature tracking. When the light is different from the left and right cameras, the average gray value of the images acquired by the left and right cameras is different, and the matching rate is low. The Fig.3(a) is a feature point distribution map obtained by FAST feature extraction on the images acquired by the left and right cameras. However, the image matching rate of the left and right cameras is not high, and the matching ratio is only 0.55. No feature points exist in the image after stereo matching. If the feature detection module does not have the feature point data output, the visual inertia mileage calculation method cannot perform the posture update, resulting in the track accumulating offset, and serious errors may occur. As shown in Fig.3(b). Uneven distribution of features. We use the distribution of feature points as variables and compare the trajectory with the original feature distribution. As shown in Fig.4(a). However, when the feature points are only distributed in the red area, the movement trajectory of the feature points is directed to the right side of the image. As shown in Fig.4(b), it can be seen that the trajectory of the feature with uneven distribution has obvious deviation to the left.
Moving feature point. The pedestrians pass in front of the camera, and the contrast track is drawn as shown in Fig.5(b). It is obvious in the circle that the green track is shifted to the left because of the influence of pedestrians. We analyzed the details of this moment. As can be seen from Fig.5(a), when the pedestrian moves, more than half of the extracted feature points are gathered on the pedestrian. Therefore, the movement of pedestrian relative to the camera will lead to the deviation of the positioning results.
In this paper, we analyzed the influences of visual measurements based on the positioning error of the system of visual inertial odometer, and analyzed the source of its positioning error. We divided the error sources of visual measurement into four cases and we have confirmed the influence of four error sources by experiments. Through the error analysis of PDR, it is found that the error of PDR in a short time is small and bounded. According to that characteristics of PDR, it is used to assist in the detection of visual errors. When the above four cases of visual errors are detected, the positioning result will be corrected.