In-Production Benchmarking for Automatic Detection of Position Errors in Indoor Localization Silas Ulrich1 , Janis Tiemann1 , Andreas Lewandowski1 and Christof Röhrig2 1 Comnovo GmbH, Robert-Schuman-Str. 6, 44263 Dortmund, Germany 2 University of Applied Sciences and Arts Dortmund, Otto-Hahn-Str. 23, 44227 Dortmund, Germany Abstract Indoor positioning systems are crucial in logistics and industrial applications for process optimization, asset tracking, safety features, and automation. However, systems that rely on dead reckoning or other sensor fusion techniques are prone to drift and require frequent recalibration. Ground truth data is often necessary to evaluate these systems, but another approach involves identifying specific features in the data stream that indicate system misbehavior. This paper presents a benchmark architecture for evaluating the performance of indoor positioning systems. This architecture identifies defined events in the data stream that indicate insufficient system performance. It also describes how to recreate the system state during events based on collected data during these events for further analysis. This approach aids in identifying root causes and expediting the development process of indoor positioning systems in real-world operational scenarios. Evaluation results show an increase in position accuracy and improved detection of system misbehavior over time when using the benchmark architecture. 1. Introduction & Related Work The deployment of Indoor Positioning Systems (IPSs) or Real-Time Locating Systems (RTLSs) in general is crucial in logistics and industrial applications for optimizing processes, tracking assets, enhancing safety, and automating operations. The accuracy and reliability of these systems are paramount, especially in environments where driver behavior impacts the positioning system, as revealed through predefined tests or simulations. In Autonomous Vehicles (AVs) research, similar challenges exist with positioning accuracy in industrial environments. Various studies evaluate AV systems through simulations and test scenarios modeled on the field of application. Thorough evaluation is essential before deploying AVs to the public due to the inefficiency and expense of public road testing, necessitating robust automated testing and simulations [1]. An example is Gomez-Huelamo et al. [2], which use a Robot Operating System (ROS) simulation layer over CARLA [3] for autonomous driving research. This approach provides insights into AV systems’ interactions with real-world conditions. The global AGV market, valued at $3.29 billion in 2019, is projected to reach $9.59 billion by 2028, growing at a Compound Annual Growth Rate (CAGR) of 12.62% [4]. In 2019, the market segments based on revenue included forklift trucks at 17.21% and pallet trucks at 13.01%, highlighting their significant impact on improving warehouse operations [4]. Mixed fleet scenarios, combining manual and automated systems, highlight the impact of manual driving on positioning systems, necessitating real-world feedback and robust testing. Recent advancements in Ultra-Wideband (UWB) technology in indoor positioning systems emphasize high accuracy and resilience. Van Herbruggen et al. [5] underscore the need for a multi-metric benchmarking approach to evaluate UWB systems, considering factors like line-of-sight conditions and algorithm selection. Their findings reveal the complexity of optimizing these systems, indicating no general solution exists. Indoor localization is often evaluated in controlled environments, potentially biasing system per- formance [6, p.12]. Real-world data is crucial for accurate system evaluation. The Microsoft Indoor Localization Competition highlighted performance variations between controlled and real-world envi- Proceedings of the Work-in-Progress Papers at the 14th International Conference on Indoor Positioning and Indoor Navigation (IPIN-WiP 2024) $ ulrich@comnovo.de (S. Ulrich); tiemann@comnovo.de (J. Tiemann); lewandowski@comnovo.de (A. Lewandowski); christof.roehrig@fh-dortmund.de (C. Röhrig) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Cloud Development Center Production Site Research & Updates Compa ny AREA 5 Feedback P P P P P P Fig. 1: Benchmarking system detecting and analyzing positioning system behavior in dynamic logistic environ- ments. ronments, emphasizing the need for comprehensive testing. Using RTLS for collision avoidance with manual vehicles in 5G environments [7] further underscores the importance of real-world testing, demonstrating how advanced technologies can enhance system robustness and safety. Sensor fusion approaches, such as those by [8], combining UWB with odometry data, simplify hardware setups. Their research on an AV car demonstrates the potential for streamlined installations, suggesting that bench- marking systems should also cover installation setups and detect unwanted behavior to improve system quality. As Fig. 1 illustrates, a system detecting and propagating information about system behavior during operation is beneficial. This includes the local machine running the position calculation and a central server for data storage and analysis, designed for real-world scenarios. The Indoor Positioning and Indoor Navigation (IPIN) challenge [9] highlights the complexity of evaluating indoor localization solutions through competitive benchmarking, suggesting diverse datasets and operator knowledge influence the evaluation process. Vedder et al. [10] demonstrate real-time testing with fault injection in AVs using UWB systems, evaluating robustness under faulty conditions. [11] emphasize realistic testing conditions in industrial scenarios, showcasing anchor placement’s importance for accuracy. Amini et al. [12] present a simulation framework converting real-world data into a simulated perception-control API, enhancing the simulation’s authenticity and underscoring real-world data’s importance. Ortega et al. [13] use rosbag 1 in ROS for capturing and analyzing log files during service robot testing. This method facilitates precise data capture, aiding performance evaluation and issue identification. Other formats include rosbag2 2 and mcap3 . In contrast to other works in the field of indoor localization, we propose an in-production systematic approach, consisting of a system architecture and an automatic method to identify potentially position error-related events. This is done using event definitions to identify such situations and collect data enabling localization engine state reproduction. Our benchmarking system runs parallel to the RTLS, propagating error event data to a central server for analysis, designed for real-world use. Current evaluation approaches using simulations or predefined tests are insufficient due to the varied logistics environments, leading to unaddressed edge cases. Our proposed benchmark architecture aims to detect and analyze system behavior during operation, speeding up development, reducing test complexity, and providing real-world feedback. 2. System Architecture To build up a benchmarking system that fits with a variety of localization engines used in logistics and industrial applications, we propose a system architecture that can be used in real-world scenarios shown in Fig. 2. It is divided into three main segments: the on-premise production site, the cloud component managing data flow, and the development interface utilizing data from the cloud for further development. The goal of this architecture is to analyze RTLS data in real-time and detect anomalies in the data stream, reducing the amount of diagnostic data to an external server for further analysis and development. The architecture is designed to be scalable and adaptable to various localization engines and sensor setups. 1 https://wiki.ros.org/Bags/Format 2 https://wiki.ros.org/Bags/Format/2.0 3 https://mcap.dev/ Cloud Dataset Ground Truth Downloader Localization Engine Data Data Management Runner Replay Map Data Anchor Data Endpoint Endpoint Industrial Vehicle Event Data Replay Detection Development Center Dataset Fleet Generator Manual Data Labeling Production Site Research & & Ticket Generation Sensor Data Other Data Dataset Data Storage Uploader Dataset Other Cloud Modules Downloader Device Software Management Machine Site Simulation Runner Industrial Vehicle Industrial Vehicle Learning Monitoring Industrial Vehicle … Simulation Vehicle Warehouse Digital Twin Digital Twin Fleet Fleet Fleet … Software Management Software Update & Release Fig. 2: System architecture of the benchmarking system, seperated into three main segments: the on-premise production site, the cloud component managing data flow, and the development interface utilizing data from the cloud for further development. The objective of this paper is to present a systematic approach to evaluate and improve system accu- racy. Various metrics can identify positional inaccuracies or faulty sensor data, revealing inconsistencies in the position stream. These events can be detected at the sensor level by analyzing the sensor value range, missing values, or large variations. Additionally, map-based collision detection with impassable objects can highlight incorrect sensor data. At a higher abstraction level, position jump detection can also indicate errors. Cross-referencing data from other high-accuracy systems, such as other forklifts, can further enhance error detection capabilities. At the production site, illustrate in Fig. 2 at the left, the localization engine operates on the vehicle, exemplified by a forklift in the diagram. This engine is an abstract representation of a RTLS that can be implemented using various technologies such as Kalman filter [14] and or other state estimators [15], which are often employed for sensor fusion in indoor positioning systems as discussed in numerous publications [16, 17, 18]. Advanced techniques may integrate sensor data with map information for localization purposes, including Simultaneous Localization and Mapping (SLAM) [19] and camera-based visual SLAM [20] methods. A ground truth system with higher accuracy than the system under test allows for performance evaluation by comparing ground truth data with system data, thus identifying discrepancies and providing direct feedback on system accuracy. In our architecture, we combine localization engine data, including incoming sensor data, with ground truth data to detect events that may indicate undesirable system behavior. The Event Detection block implements interrupt functions triggered based on the sensor set and localization technique parameters. Upon event detection, the data necessary for event recreation, including raw sensor data, optional ground truth data, and system state information, is compiled into a dataset. This dataset is then transmitted to a backend by the Data Uploader for further analysis. By targeting specific error-related data through the Event Detection functions, the volume of data gathered for testing can be minimized, focusing on the essential information needed for accurate error analysis. The cloud component manages data flow and provides an interface for development. Data from multiple vehicles is uploaded to the cloud, where it is processed by various modules, including data endpoints, data management endpoints, and other cloud modules such as Machine Learning, Site Monitoring, Vehicle Digital Twin, and Warehouse Digital Twin. Fig. 2 shown on the right, the Replay Runner is utilized by a human operator who labels event scenarios. An operator examines the data to identify false positives that can be ignored and detects issues that require development attention. If an issue is identified, the operator generates a ticket and links the relevant data for the development team to address. The Simulation Runner is used for an in-depth analysis and debugging of scenarios. It provides detailed outputs to help identify and fix issues. Once a software fix is validated in the simulation environment, the update is remotely deployed to the vehicle or fleet. This ensures that the software running on the vehicles is continuously improved based on real-world data and feedback. This architecture creates a robust feedback and update loop. Initially, the localization engine detects events at the production site, and the relevant data is collected and uploaded to the cloud. The Replay Runner allows for human intervention to label and categorize the events, generating tickets for issues that need development. The Simulation Runner then enables a detailed analysis and debugging process, facilitating the identification and resolution of issues. Once fixes are made, software updates are deployed back to the vehicles, ensuring the system evolves and improves continuously. By integrating these components, the architecture supports robust error detection and system performance analysis, facilitating continuous improvement in localization accuracy and reliability. 3. Event Detection Our localization engine is based on a filter that gathers sensor data from a forklift and ranging mea- surements from a sparse UWB system using Two-Way Ranging (TWR). The vehicle is equipped with a computing unit connected to the sensors via a Controller Area Network (CAN) bus. We equipped a reference-vehicle with a Commercial Off-The-Shelf (COTS) Ground Truth (GT) system. The GT system provides accurate position data, which is used to evaluate the performance of the localization engine. It is connected to the vehicles’s computing unit via Ethernet, providing real-time position data to the localization engine. For the event detection several techniques are used to detect errors in the localization engine. The section is split into parts, where the first is dedicated to a setup that can be used without ground truth data available on vehicles, especially in our scenario where an estimating filter is used for localization. The second part is dedicated to the use of ground truth data to detect errors in the localization engine. 3.1. Techniques for Internal Localization Engine Detection Given a sensor fusion system like our filter, having vehicle and UWB data, the following techniques can be used to detect errors in the localization engine. Sliding Window of Sensor Values and Speed Influence A significant increase in the standard deviations of position 𝑝 = (𝑥, 𝑦)⊺ and orientation (𝜃) can indicate potential errors in the localization engine. Implementing a sliding window technique ensures consistent error detection and mitigation over time. By concurrently monitoring orientation and position variance, the system can more accurately identify potential errors. For each time step 𝑡, the standard deviations of the position and orientation of the state estimations are calculated as follows: ⎯ ⎸ ⎸ 1 ∑︁𝑁 𝜎𝑝,state,𝑡 = ⎷ ((𝑥𝑖 − 𝜇𝑥,𝑡 )2 + (𝑦𝑖 − 𝜇𝑦,𝑡 )2 ), (1) 𝑁 𝑖=1 ⎯ ⎸ 𝑁 ⎸ 1 ∑︁ 𝜎𝜃,state,𝑡 = ⎷ (𝜃𝑖 − 𝜇𝜃,𝑡 )2 . (2) 𝑁 𝑖=1 Subsequently, a sliding window is applied to these standard deviations: 𝑤−1 𝑤−1 1 ∑︁ 1 ∑︁ 𝜇𝜎𝑝 ,𝑡 = 𝜎𝑝,state,𝑡−𝑖 and 𝜇𝜎𝜃 ,𝑡 = 𝜎𝜃,state,𝑡−𝑖 . (3) 𝑤 𝑤 𝑖=0 𝑖=0 The conditions for detecting potential errors are defined as instances when the sliding window mean standard deviations of position and orientation exceed predefined thresholds: 𝜇𝜎𝑝 ,𝑡 > 𝜎𝑝,threshold ∧ 𝜇𝜎𝜃 ,𝑡 > 𝜎𝜃,threshold . (4) These equations represent the methodology for calculating the standard deviation of position and orientation for state in our filter, followed by the application of a sliding window to these standard deviations. It should be noted, that estimators typically already provide such a metric such as covariance- matrix, or belief that could be utilized after or without some form of sliding window filtering. Based on these results, the system flags potential errors when the computed sliding window standard deviations and the vehicle speed exceed predefined thresholds. Plausibility Checks Filter Incorporating plausibility with our filter can enhance accuracy by eliminating colliding state estimates with pre-known non-plausible ones. However, due to odometry drift, extensive non-plausible state depletion or collisions leading to total state loss can provide insights into position reliability. By monitoring the ratio of remaining state 𝑁current estimates after a plausibility check to the total count 𝑁max , error scenarios can be identified. The deletion rate of the current state count can be calculated as: 𝑁max − 𝑁current Δrate = . (5) 𝑁max This deletion rate Δrate is then added to the total state ratio deleted since the start: 𝑇 ∑︁ Δtotal = Δrate,𝑡 . (6) 𝑡=0 The total deletion ratio provides a cumulative measure of non-plausible state loss over time, which is indicative of the system’s position reliability. It is important to note that this total deletion ratio is reset whenever a UWB position calculation is performed. This reset ensures that our filter can reinitialize with fresh data, thereby maintaining the accuracy and reliability of the localization system. By systematically monitoring these metrics, the system can identify potential error scenarios and make necessary adjustments to improve position reliability. Sensor Calibration Age Vehicle data, when used for odometry calculations, necessitates in-operation calibration to mitigate drift. This approach is akin to the one employed in Pedestrian Dead Reckoning (PDR) [21, 22], and can be similarly applied to vehicle localization systems. By integrating the last calculation time into error detection, it becomes feasible to correlate potential inaccurate measurements with drift caused by inadequate calibration. This correlation is represented by 𝜅𝛾 , with thr serving as the threshold for calibration age. ⎯ ⎸ 3 (︂ )︂2 ⎸∑︁ 1 1 𝜅𝛾 = ⎷ − , (7) 𝛾measured,𝑗 𝛾bias,𝑗 𝑗=1 𝜅𝛾,last > 𝜅thr . (8) Time Since Last Reliable Measurement In parallel to the calibration detection 𝜅𝛾 , the time since the last absolute position calculation based on UWB range measurements can be used to detect potential errors. This time is represented by Ωlast , where Ω denotes the time of the last reliable UWB measurement. The threshold for the maximum allowable time since the last reliable UWB measurement is represented by Ωthr with Ωlast > Ωthr . (9) Discrepancy Detection in Position Updates upon UWB Zone Entry Abrupt variations in position updates, especially during the transition from relative to absolute posi- tioning via UWB measurements, may signify potential inaccuracies. Such discrepancies, surpassing a predetermined threshold associated with the minimum accuracy requisites of the RTLS, warrant further scrutiny. The positional change at each time step 𝑡 is computed as the absolute difference between the current position p𝑐𝑢𝑟𝑟 and the position at the most recent UWB measurement pUWB,last : Δpos = |p𝑐𝑢𝑟𝑟 − pUWB,last | . (10) Should this positional change surpass a predetermined threshold, it necessitates further examination: Δpos > Δthreshold . (11) Moreover, the surveillance of abrupt orientation alterations is advantageous, as they can be indicative of implementation inaccuracies. 3.2. Techniques for Ground Truth-Based Error Detection Having access to accurate GT data allows to rate the system accuracy on a position level. The following techniques can be used to detect errors in the localization engine based on the GT data. Position Offset Detection The Euclidean distance between the estimated and GT positions serves as a measure of localization error. This metric is particularly useful in assessing the installation quality of UWB setups in UWB enabled areas, as factors such as improper mounting or obstructive objects can result in erroneous measurements. Furthermore, the GT provides a valuable benchmark for evaluating the parameterization of the sensor fusion system, supplementing other error metrics to isolate and identify specific components of the sensor fusion system that may be contributing to errors. (12) √︀ 𝑑Euclidean = (𝑥GT − 𝑥est )2 + (𝑦GT − 𝑦est )2 . This can be extended to the orientation error between the estimated and GT positions as well. (13) ⃒ ⃒ 𝜃error = ⃒𝜃filter − 𝜃gt ⃒ . Mahalanobis Distance for State-Estimate Inclusion The Mahalanobis distance provides an error metric that is more related to the filter by considering the entire state estimate distribution. Unlike the Euclidean distance, which only measures the direct spatial offset between the estimated and GT positions, the Mahalanobis distance takes into account the spread and shape of the state distribution. This metric measures how many standard deviations away the GT position is from the mean of the state-estimate distribution, thereby offering a deeper insight into the system’s accuracy and is calculated as follows: √︁ 𝑑Mahalanobis = (𝑥GT − 𝜇𝑥 )𝑇 𝑆 −1 (𝑥GT − 𝜇𝑥 ). (14) Where 𝜇𝑥 is the mean vector of the state positions and 𝑆 is the positive-definite covariance matrix of the state estimate positions. By focusing on the state-estimates, the Mahalanobis distance accounts for the distribution and variance within a filter, providing a more robust metric for error detection in comparison to the Euclidean distance. This approach is particularly useful for identifying outliers and assessing the overall reliability of the estimated position relative to the ground truth when the state-estimate-based position is used during unsupported estimation. 4. Experimental Results Speed Deletion Rate Window of Theta Var Speed Deletion Rate Window of Theta Var Window of Speed Var Window of Pos Var Theta Window of Speed Var Window of Pos Var Theta Speed [m/s] µσv2 ,t [m/s] Speed [m/s] 2.5 µσv2 ,t [m/s] 2.5 1.0 1.0 0.0 0.0 0.0 -2.5 0.0 5.0 ∆rate [%] ∆rate [%] 5.0 0.0 0.0 µσp [m] µσp [m] 5.0 2.5 0.0 0.0 0.1 2.5 2.5 µσφ [rad] µσφ [rad] φ [rad] φ [rad] 0.0 0.05 0.0 0.0 -2.5 0.0 -2.5 60.0 120.0 180.0 240.0 300.0 360.0 420.0 480.0 540.0 60.0 120.0 180.0 240.0 300.0 360.0 420.0 Ωlast ∆pos [m]: 9.55 ∆pos [m]: 4.00 Ωlast ∆pos [m]: 0.23 t [s] t [s] (a) Sensor data 1 (b) Sensor data 2 Fig. 3: (a) and (b) depict sensor data for two distinct runs. The shaded regions represent UWB measurements, while the black vertical lines signify the commencement of unsupported estimation. The red lines denote the position difference Δpos , instances where relative positioning is rectified using absolute measurements. The window size 𝜇 is set to 5 s. In the given scenario, sensor data is passively collected from multiple vehicles operating on-site during the production process, as detailed in Section 3. The dataset, representing one month of data collection from multiple vehicles at a single site, comprises a total of 4,129,602 position events. Each position event encapsulates a timestamp, current speed, pose information (𝑥, 𝑦, 𝜃), and the standard deviation (𝜎) for each value. Additionally, the dataset includes the vehicle’s speed and the current state-estimate depletion ratio. The illustration in Fig. 3 presents two selected traces that underscore the data. A more detailed analysis of the data reveals several insights. Firstly, the magnitude of the discrepancy is not readily discernible by merely observing the quantity of state-estimate wipe, standard deviation of position, and theta. Secondly, alterations in theta and speed influence the standard deviation of position. Finally, based on the map settings at the production site, in narrow corridors, the jump size is constrained to the width of the corridor. Consequently, errors may occur even when jumps are relatively small, underscoring the necessity of a combination of error detection methods. Utilizing the dataset provided, the benchmark pipeline is used to manually processes the data to ascertain the thresholds for the event detection methods, as described in Section 2. Moreover, datasets associated with localization engine issues are employed in the simulation segment of the benchmark pipeline. Fig. 4 shows the monthly event count of a real production site, having multiple manually driven vehicle in operation. Because this is a work in progress, thresholds are manually picked and not validated in combination with the ground truth data. The presented data shows an example of one RTLS system in a real-world scenario. During the dataset generation multiple software changes were made to the localization engine settings, resulting in a variation of event counts. The link between event count and adaptions in the localization engine settings is visible in the data. To have this evaluated in a controlled test scenario, we plan to test different loaclization settings during specific time frames, resulting in a variation of event counts. This will be validated with a ground truth system to ensure the thresholds are set correctly. 5. Conclusion & Discussion This paper presented a preliminary architecture for in-production benchmarking of indoor localization systems, focusing on real-time error detection through event-driven data collection. While the archi- tecture shows promise in identifying system misbehaviors, it currently relies on manually calibrated thresholds, which have not been validated against ground truth data. As such, the detection accuracy is not yet reliable enough for fully autonomous operation. ×106 Total Data Seconds and Events by Month 1.75 12000 1.50 10000 1.25 Total Datapoints Total Events 8000 1.00 6000 0.75 4000 0.50 0.25 2000 0.00 0 04 05 06 − − − 24 24 24 20 20 20 Jump Events Theta Std Events State Deletion Events Position Std Events Speed Change Events Fig. 4: Total amount of events by category and datapoints for the given production site by month. The categories are calculated over a slinding window of 5 seconds. The thresholds are manually set: 45∘ for the theta standard deviation, 5m for the position standard deviation, 2.77m s−1 for the speed change, 10m for the jump event, and 60% for the state deletion event with the plausibility check. The next steps involve refining these thresholds using ground truth data to enhance the system’s real-time capabilities. Additionally, validation against established benchmarking systems is essential to assess the overall effectiveness of the system. We invite the research community to explore questions such as: How can this system be validated and compared against existing methods in diverse operational environments, especially when running in operational scenarios? These discussions are crucial as we aim to create a more robust, scalable solution for real-world industrial applications. By addressing these challenges collaboratively, we can significantly advance the reliability and performance of indoor localization systems. References [1] H. Alghodhaifi, S. Lakshmanan, Autonomous Vehicle Evaluation: A Comprehensive Survey on Modeling and Simulation Approaches, IEEE Access 9 (2021) 151531–151566. doi:10.1109/ ACCESS.2021.3125620, conference Name: IEEE Access. [2] C. Gómez-Huélamo, J. Del Egido, L. M. Bergasa, R. Barea, E. López-Guillén, F. Arango, J. Araluce, J. López, Train here, drive there: ROS based end-to-end autonomous-driving pipeline validation in CARLA simulator using the NHTSA typology, Multimed Tools Appl 81 (2022) 4213–4240. doi:10.1007/s11042-021-11681-7. [3] C. Team, CARLA, 2024. URL: http://carla.org//. [4] A. Thakur, A Conceptual Market Analysis of Automated Vehicles for Logistics in Future, Journal of Supply Chain Management Systems 11 (2022) 24. [5] B. Van Herbruggen, J. V.-V. Gerwen, S. Luchie, Y. Durodié, B. Vanderborght, M. Aernouts, A. Munteanu, J. Fontaine, E. D. Poorter, Selecting and Combining UWB Localization Algo- rithms: Insights and Recommendations From a Multi-Metric Benchmark, IEEE Access 12 (2024) 16881–16901. doi:10.1109/ACCESS.2024.3358274. [6] D. Lymberopoulos, J. Liu, The Microsoft Indoor Localization Competition: Experiences and Lessons Learned, IEEE Signal Process. Mag. 34 (2017) 125–140. doi:10.1109/MSP.2017.2713817. [7] S. Ulrich, T. Luong, C. Moldovan, J. Tiemann, A. Lewandowski, C. Röhrig, System Architecture for Digital Twin Based Collision Avoidance Through Private 5G Networks, in: 2023 IEEE 12th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), IEEE, Dortmund, Germany, 2023, pp. 199–204. doi:10. 1109/IDAACS58523.2023.10348730. [8] D. Grzechca, A. Ziębiński, K. Paszek, K. Hanzel, A. Giel, M. Czerny, A. Becker, How Accurate Can UWB and Dead Reckoning Positioning Systems Be? Comparison to SLAM Using the RPLidar System, Sensors (Basel) 20 (2020) 3761. doi:10.3390/s20133761. [9] F. Potorti, P. Barsocchi, M. Girolami, J. Torres-Sospedra, R. Montoliu, Evaluating indoor localization solutions in large environments through competitive benchmarking: The EvAAL-ETRI competition, in: 2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN), IEEE, Banff, AB, Canada, 2015, pp. 1–10. doi:10.1109/IPIN.2015.7346970. [10] B. Vedder, B. J. Svensson, J. Vinter, M. Jonsson, Automated Testing of Ultrawideband Positioning for Autonomous Driving, Journal of Robotics 2020 (2020) 1–15. doi:10.1155/2020/9345360. [11] A. Schjørring, A. L. Cretu-Sircu, I. Rodriguez, P. Cederholm, G. Berardinelli, P. Mogensen, Per- formance Evaluation of a UWB Positioning System Applied to Static and Mobile Use Cases in Industrial Scenarios, Electronics 11 (2022) 3294. doi:10.3390/electronics11203294. [12] A. Amini, T.-H. Wang, I. Gilitschenski, W. Schwarting, Z. Liu, S. Han, S. Karaman, D. Rus, VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles, in: 2022 International Conference on Robotics and Automation (ICRA), IEEE, Philadelphia, PA, USA, 2022, pp. 2419–2426. doi:10.1109/ICRA46639.2022.9812276. [13] A. Ortega, N. Hochgeschwender, T. Berger, Testing Service Robots in the Field: An Experience Report, in: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2022, pp. 165–172. doi:10.1109/IROS47612.2022.9981789. [14] R. E. Kalman, A New Approach to Linear Filtering and Prediction Problems, Journal of Basic Engineering 82 (1960) 35–45. doi:10.1115/1.3662552. [15] J. S. Liu, R. Chen, Sequential Monte Carlo Methods for Dynamic Systems, Journal of the American Statistical Association 93 (1998) 1032–1044. doi:10.1080/01621459.1998.10473765. [16] F. Zafari, A. Gkelias, K. K. Leung, A Survey of Indoor Localization Systems and Technologies, IEEE Communications Surveys & Tutorials 21 (2019) 2568–2599. doi:10.1109/COMST.2019.2911558, conference Name: IEEE Communications Surveys & Tutorials. [17] R. F. Brena, J. P. García-Vázquez, C. E. Galván-Tejada, D. Muñoz-Rodriguez, C. Vargas-Rosales, J. Fangmeyer, Evolution of Indoor Positioning Technologies: A Survey, Journal of Sensors 2017 (2017) 1–21. doi:10.1155/2017/2630413. [18] L. Mainetti, L. Patrono, I. Sergi, A survey on indoor positioning systems, in: 2014 22nd International Conference on Software, Telecommunications and Computer Networks (SoftCOM), IEEE, Split, Croatia, 2014, pp. 111–120. doi:10.1109/SOFTCOM.2014.7039067. [19] H. Durrant-Whyte, T. Bailey, Simultaneous localization and mapping: part I, IEEE Robotics & Automation Magazine 13 (2006) 99–110. doi:10.1109/MRA.2006.1638022, conference Name: IEEE Robotics & Automation Magazine. [20] K. Yousif, A. Bab-Hadiashar, R. Hoseinnezhad, An Overview to Visual Odometry and Vi- sual SLAM: Applications to Mobile Robotics, Intell Ind Syst 1 (2015) 289–311. doi:10.1007/ s40903-015-0032-7. [21] W. Zhang, X. Li, D. Wei, X. Ji, H. Yuan, A foot-mounted PDR system based on IMU/EKF+HMM+ZUPT+ZARU+HDR+compass algorithm, in: 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), IEEE, Sapporo, 2017, pp. 1–5. doi:10.1109/ IPIN.2017.8115916. [22] A. R. Jimenez Ruiz, F. Seco Granja, J. Carlos Prieto Honorato, J. I. Guevara Rosas, Pedestrian indoor navigation by aiding a foot-mounted IMU with RFID Signal Strength measurements, in: 2010 International Conference on Indoor Positioning and Indoor Navigation, IEEE, Zurich, Switzerland, 2010, pp. 1–7. doi:10.1109/IPIN.2010.5646885.