In indoor localization, visual-inertial odometry (VIO) is widely used due to its low cost and efective environmental perception capabilities. However, feature based VIO performs poorly in indoor pedestrian localization challenges. The feature points are prone to occlusion or tracking loss during complex pedestrian movements, and IMU is dificult to perform dead reckoning due to irregular or excessively large anomalous measurements. To address these issues, this paper proposes a VIO approach using the direct method assisted by a neural inertial network. The proposed method replaces the feature based method in front-end in VIO with the direct method to mitigate issues related to occlusion and tracking loss of feature points. It utilizes a neural inertial network to supply initial values for pixel tracking within the direct method and and integrates dead reckoning results as constraints during position optimization. Experimental results demonstrate that the method proposed in this paper exhibits higher positioning accuracy and robustness compared to existing methods in indoor pedestrian localization scenarios.
As a component of navigation, positioning, and timing technologies, indoor positioning technology is
widely applied in fields such as indoor robot navigation, augmented reality, the Internet of Things, and
indoor rescue. However, unlike outdoor positioning, GNSS signals are unavailable in indoor scenarios,
making it more dificult to obtain accurate and robust positioning results. Most indoor positioning
solutions typically use Bluetooth, UWB, or WiFi [
However, VIO does not perform optimally in indoor pedestrian localization. Most VIO systems currently employ feature points in visual processing to recover the relative position transformation between image frames. Nevertheless, when pedestrians carry the camera in motion, large changes in the image field of view or occlusions can lead to feature point loss, resulting in system failure. In the inertial component, consumer-grade IMUs are plagued by biases, noise, and thermal drift, resulting in significant measurement errors. Additionally, due to the highly irregular nature of pedestrian movement, position estimation based on kinematics or gait tracking is also suboptimal.
Therefore, to address the problems in indoor pedestrian localization, we propose a VIO system assisted
by a neural inertial network to enhance the accuracy and robustness. The method employs a direct
method within the VIO framework based on DM-VIO [
Visual-inertial navigation systems (VINS) are typically derived from visual SLAM methods integrated
with inertial sensors. The incorporation of inertial measurement data introduces stable scale information
to SLAM, thereby efectively enhancing the accuracy of image inter-frame matching. Most current
methods achieve tightly coupled VIO by fusing raw image and IMU data. The most classic approach is
the Multi-State Constraint Kalman Filter (MSCKF) [
With the improvement in computational performance, VO methods can process more pixel data in
realtime, leading to the emergence of direct VIO methods, which difer from feature point based approaches.
Direct methods operate directly on the image pixels captured by the camera, thus eliminating the
need for stable extraction and matching of feature points in the environment. The Direct Sparse
Odometry (DSO) algorithm [
The emergence of neural inertial network methods enables deep analysis of IMU measurement data,
extracting underlying motion patterns in complex movements. The RIDI [
In VIO methods, the processing of IMU data still adopts the method of integration using kinematic models in traditional navigation methods. In order to decrease the computational load and efectively improve the accuracy of inter-frame relative position estimation, the integration is typically transformed into an inter-frame pre-integration that remains unafected by the initial state. However, these kinematic models perform poorly in complex motion scenarios, as irregular or excessive IMU measurements can cause the system state to gradually diverge.
Some algorithms have incorporated position prediction results from inertial neural networks into
combined navigation systems, but fewer of them add inertial neural networks to VIO. TLIO [
However, it is worth noting that all of the above methods only involve the neural network inference results as additional constraints added to the feature point based VIO. This structure leads to neural network inference results with less impact on such systems. In contrast, the direct methods, although more accurate, require higher IMU data quality. Bad IMU data will cause the direct method to fail to initialize properly and have poorer results, which has a stronger dependence on IMU data.
Therefore, we implement a neural inertial navigation aided method VIO. The proposed method is the first method to combine inertial neural networks with direct method VIO, and it combines inertial neural networks more tightly than the existing methods. The method efectively improves the accuracy and robustness of the direct VIO initialization and solution.
boxes in Fig. 1.
The proposed method in this paper is a direct visual-inertial odometry, which improves upon the
DM-VIO framework, as illustrated in Fig. 1. After the visual part completes initialization and the
neural inertial navigation accumulates suficient IMU measurement data, the network will estimate
the position of carrier. The method utilizes the network’s predicion results to set an initial relative
position between image frames, providing a better initial photometric error for the visual part. In the
joint optimization phase, the prediction results are also integrated into the factor graph as a constraint
for position estimation. Compared to DM-VIO, the improvements of our method are highlighted in red
In direct VO [
In DM-VIO [
We utilize the neural inertial network and proposed in [
is the weight related to the gradient. ′ is the
DM-VIO incorporates photometric errors and IMU residuals for optimization process. And its energy function is
= ℎ + which consists of the photometric error ℎ, pre-integration residuals . And is the weight ︁( ∆ ′ , ′ )︁ = (︀ (− , − ), . . . , (, ), ℎ−
︀) = ( − ) − = ( − ) ∆ ′ and its covariance ′ are the outputs of the network. where (∙ ) is the function obtained by neural network fitting. and are the raw acceleration and angular velocity measured by the IMU sensors. and are the bias obtained from the VIO system. is the gravity vector. is the VIO coordinate system while is the IMU coordinate system. and are the acceleration and angular velocity at the th IMU time step in VIO coordinate system, respectively. ℎ is the hidden state produced by LSTM at the last time step. The relative displacement 3.4. NIN-DSO
To ensure that the predicion results of network facilitate efective convergence of the VIO position estimation, we bundle these predictions to the image inputs for approximate temporal synchronization. Indeed, predictions from the neural inertial network resemble IMU pre-integration, serving as a type of constraint between images. Therefore, the proposed method adjusts the prediction output frequency to match the images input frequency. Additionally, in order to minimize the temporal discrepancy between the images and the predition results, we interpolate the network inference results at the image input moments. The temporal relationship of IMU data, image data, and network prediction results is illustrated in Fig. 2.
as shown in (4).
In this paper, we use the relative position transformation predicted by neural inertial networks as an alternative for initializing the photometric error, providing greater robustness than IMU pre-integration, = ∑︁ ||( [︀ ′ − ) − ( [] − ) / ︀] || ∈ items are consistent with (1). where ′ is the projected point obtained by prediction results of network. And the remaining
The VIO proposed in this paper achieves position estimation by minimizing the energy function , which comprises photometric error, pre-integration residuals, and network prediction position residuals. Unlike the feature point based method, our approach simultaneously optimizes the position, sensor error, scale and 3D structure, resulting in higher accuracy. The factor graph for the proposed method is shown in Fig. 3.
Based on DM-VIO, the method further incorporates the position predictions from the neural inertial network as additional constraints. And its energy function is
= ℎ + + which consists network prediction position residuals . The photometric error is the sum of the individual pixel photometric erros, and can be determined by (6).
ℎ = ∑︁ ∑︁ ∑︁
∈ ∈ ∈() where is the set of keyframes . is the set of points in keyframes. And () is the set of points that are jointly observed in keyframes. (4) (5) (6) ︁( ⊙ ˆ ︁) * ∑︁ − 1 , * ︁( ⊙ ˆ ︁) corresponding covariance of the state. And ⊙ is the subtraction operation in Lie algebra. where is the state obtained from pre-integration. ˆ is the estimated state of VIO. ∑︀ , is the
The IMU measurements are preprocessed before being input into the network in our method. As a
result, the relative position transformation predicted by the network remains consistent with the VIO
coordinate system. Based on this characteristic, in the part of neural inertial navigation, we use the
predictions as direct constraints between image frames rather than the constraints in [
=︁( ⊙ ˆ ︁) * ∑︁ − 1 , *
⊙ ︁( ˆ ︁) corresponding covariance of the state. where
is the state obtained from Neural inertial network prediction results. ∑︀, is the
This section presents the test results on various datasets to demonstrate the efectiveness of our method. with Ubuntu 20.04 and NVIDIA GeForce RTX3060 12GB graphics card.
All the experiments were conducted on an Intel Core I5-10510U CPU@1.80GHz× 8 computer equipped
In this subsection, we evaluate the performance of the proposed method using the TUM-VI dataset
[
From the data in TABLE 1, it is evident that the position estimation of direct method based VIO outperform feature point based method, regardless of whether an neural inertial network is added. This indicates that direct method VIO is more suitable for indoor pedestrian positioning scenarios. Our method achieved the best results across all data in the room sequence, reducing the average ATE by 60.2% compared to DM-VIO. Notably, in the fourth data set, the ATE was reduced by 84.2%, demonstrating that the addition of neural network predictions efectively addresses the issue of DM-VIO’s inability to track and converge efectively when IMU data quality is poor. (7) (8)
Compared to the room sequence of the TUM-VI dataset, indoor pedestrian positioning typically involves faster movement speeds, more complex lighting environments, and larger movement scales. Additionally, VIO tends to accumulate significant errors over long periods of operation. Therefore, we collected a custom dataset to supplement the experiments. This dataset was collected using a Flir camera and a Livox Avia LiDAR as the acquisition sensors, with IMU measurements provided by the Avia. The frequency of image data is 10Hz and the resolution is 1440× 1080. The frequency of IMU data is 200Hz. And the two sensors are synchronized by hardware trigger. The pseudo ground truth was obtained from a higher precision Lidar-Visual-Inertial Navigation System to evaluate the performance of the method. In this dataset, we carried the device around the interior of the New Main Building at Beihang University. The total distance covered is 513.56 meters, and the duration of the data collection lasts 368.41 seconds. The data contains typical pedestrian positioning characteristics, and an example of large attitude change and complex lighting environment from the data is shown in Fig. 4.
As with the experiments in the first subsection, the performance of the algorithms is still evaluated using ATE. The comparison of the results among VINS-Mono, NIN-VINS, DM-VIO, and NIN-DSO are shown in TABLE 2.
From the TABLE 2, it can be seen that our method still performs well on this custom data, achieving a 50.7% reduction in ATE relative to the best method among the other three. The same conclusion can be drawn from experiments on the public dataset. Direct methods are more suitable for indoor pedestrian localization than feature point based methods. Additionally, incorporating prediction of neural inertial network can efectively address issues where the system fails to track and converge properly, mitigate cumulative errors during long term positioning, and consequently improve the accuracy and robustness of VIO.
The estimated trajectories and ATE curves for diferent method on this dataset are shown in Fig. 5, wiht the starting points of these trajectories marked with triangles. As can be seen in Fig. 5, the accuracy using the direct method is higher than the feature point method. In the case of starting positions with similar errors, our method has better scaling in the middle of the trajectory compared to DM-VIO.
(a) The trajectories of diferent method.
(b) The ATE curve of diferent method.
This paper proposes a direct visual-inertial odometry with the assistance of a neural inertial network. The neural network is implemented and trained using PyTorch, and deployed within the VIO framework utilizing ONNX and TensorRT. The inclusion of neural inertial navigation provides better initial values for the image inter-frame tracking in direct method based VIO, leading to a more stable visual initialization process than DSO and DM-VIO. Furthermore, by incorporating the network’s position prediction results into the energy function of the optimization model, in addition to photometric error and IMU residuals, the system reduces its reliance on the visual component in complex motion environments and mitigates errors and divergence of IMU data. Experimental results demonstrate that the proposed neural inertial navigation aided direct VIO achieves more accurate and robust position estimation in indoor pedestrian movement scenarios.
Although the proposed method does not require stable feature point extraction and matching as feature based methods, it relies on visual component as the foundation for system calculations, rendering it unable to estimate positions when the visual component fails. The current supplementation using the neural inertial network does not fundamentally solve the problem. We plan to further leverage the powerful data processing capabilities of neural networks to evaluate visual inter-frame position estimation results using IMU data. By using this approach, we aim to reduce the reliance of the direct VIO on the visual component, enabling the system to continue functioning normally even when the visual component fails.
The project is supported by the National Natural Science Foundation of China (Grant No. 42274037), the Aeronautical Science Foundation of China (Grant No. 2022Z022051001), and the National Key Research and Development Program of China (Grant No. 2020YFB0505804).