An Approach to Skid Detection Using Optical Flow and IMU Sensor Nikola Totev 1 and Boris Robev 1 1 Faculty of Mathematics and Informatics, University of Sofia “St.Kliment Ohridski“, James Bouchier blvd., 1164, Sofia, Bulgaria Abstract Estimating the position of mobile robots is a key part of mobile robots performance and productivity and it is based on the different localization methods and sensors to be used. The goal of this paper is to propose a method of accurately estimating the position of a mobile robot using an optical flow sensor together with an IMU (Inertial Measurement Unit) sensor and encoders. Using these three sensors for observing and monitoring the movement of the robot, most expected and unexpected movements can be accounted for as well as a good pose estimation can be performed more accurately. The paper proposes an approach to prevent skidding of mobile robots and mitigate and avoid such skidding with an appropriate localization and position estimation modeling of mobile robots that uses multiple and different sensors. All sensors track and measure overall velocity, angular velocity, orientation and positioning, as well as acceleration of the robot. A localization algorithm is based on the probabilistic localization methods and Kalman filter algorithm. Keywords Skid detection, optical flow, sensor fusion, Kalman filter, SLAM 1. Introduction Robots can be put into one of two groups, industrial and service. According to the International Federation of Robotics (IFR) – “Service robots are technical devices that perform tasks useful to the well-being of humans in a semi or fully autonomous way” [1]. The IFR applies the definitions and standards for service robots according to ISO 8373:2012 [4]. In some cases, there are details that ac- cording to IFR definitions are not applicable or lead to the specific ambiguous definition of different categories of robots from the generally accepted standard. To distinguish industrial robots from service robots, IFR accepts ISO criteria de- Information Systems & Grid Technologies: Fifteenth International Conference ISGT’2022, May 27–28, 2022, Sofia, Bulgaria EMAIL: nikolart@uni-sofia.bg (N. Totev); robev@fmi.uni-sofia.bg (B. Robev) ORCID: 0000-0001-9220-0322 (N. Totev); 0000-0002-2705-3941 (B. Robev) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) termining their application in industrial automation to non-industrial automation as sufficient to classify a robot as an industrial or service robot. The main difference between service and industrial robots is the area of their application and service domains, and the closeness to the end user [2]. Because service robots operate much closer to the end user, there are some important aspects of the design such as safety, reliability, cost, appearance, and user inter- face [3] that need to be considered. Considering this classification, service robots mainly provide services for professional and non-professional usage, namely cat- egorization by application criteria. Second type categorization of service robots is by their movement on a different surface: ground and hard surface, water surface, aerial movement, and others [8, 9]. Another important aspect of service robots is that they work in unstructured and uncertain environments. This means service robots should be designed to safely handle uncertainty while operating. One such uncertainty is position esti- mation/localization, as the proper localization of service robots at any moment of their operation is essential for the quality of service provided by them. Inac- curate estimation of the position of service robots will cause loss of tracking and create uncertainty in the robot’s activities. These inaccuracies in the position estimation of mobile robots can be caused by some kind of skidding caused by insufficient traction between the wheels and the terrain the robot is operating on, or unexpected lateral shifts or rotations caused by external forces. If not detected and corrected these disturbances will lead to erroneous localization data, which in turn will affect the mapping data in a negative way. The MIRACle project (https://miraclebg.com) Intelligent Urban Environ- ment Laboratory (IUE-Lab) concept is based on deployment service robot’s models for mass usage using registering, analysis and storing observation data from environments, where the robots are operating. The goal is to collect and process all receiving data from intelligent sensors located on the service robot’s body, in order an appropriate model for autonomous service to be delivered to the consumers. In this case, the integration between sensors and pure mechanical part could be used in one of the functional sub-groups of IUE, named Intelligent Home Environment (IHE). Deployment of IHE are based on layered structure, including: infrastructure, data and service layers. This paper proposes a localization and position estima- tion approach for skid detection, that uses multiple different sensors to detect and compensate for the unexpected skidding movements described. The proposed ap- proach includes design system architecture for mobile service robot with optical flow sensor, IMU sensor and encoders; probabilistic method for localization and positioning; localization and data flow model for future deployment in IHE. 204 2. Localization methods This chapter focuses on the localization methods and how different sensors can come together to create a robust approach. The different sensors discussed in this paper are optical flow sensors, IMU modules, and encoders. A further model and analyzing how the strongest features of each sensor can be used to create a robust localization model are covered in later chapters. Accurate position estimation is a key component to the successful operation of most autonomous mobile robots. In general, there are three phases that comprise the motion of a mobile robot: localization, path planning, and path execution.[4] Two main methods are used in mobile robot localization, based on two dif- ferent perspectives: probabilistic methods and autonomous map building meth- ods. The first method uses the Kalman filter (KF) algorithm, It requires the pres- ence of an environment map, as in most cases, the map is predefined, including all surrounding objects, obstacles, walls, and trajectories of the robot’s movement. In a non-dynamic environment, the localization could be simple as the robot is the only object moving on the surface. The information acquired from mobile robots is combined together to estimate the actual position and localization of the robot, as achievement of their optimal estimation will avoid mobile robots posing uncertainty. In this regard, the algorithm allows combining uncertainties The second method, simultaneous localization and mapping (SLAM), use Extended Kalman Filter (EKF) algorithm for automatic map construction during mobile robot localization. In a dynamic environment, localization of the robot is more difficult, because the robot movement depends on the movement of other moving objects surrounding the robot. In such cases, the configuration of the environment changes due to the presence of dynamic objects or explicit modifi- cation. This problem is solved by automatic map building: the robot starts navi- gating from a random initial point. It explores the environment autonomously through its sensors. Further, it gains environmental knowledge and builds a map by interpreting the scene. This helps the robot to localize relative to the map [5]. SLAM also uses a probabilistic method, where the robot’s position and robot map is estimated. 3. System architecture Following the definition and layered structure of IHE, the infrastructure layer is based on integrated sensors, local data concentrators, server and/or data col- lecting and processing equipment and special actuators, depending of the purpose of service robots. The proposed approach uses 3 different sensors to perform position estima- tion. Along with these sensors, a Raspberry PI is used to process camera data by 205 performing an optical flow algorithm and store sensor data from the encoders and IMU a block schema of the platform architecture can be seen in Figure 1. The encoders are based on an IR reflectance sensor and are mounted near the wheels. Slotted discs are used to interrupt the IR beam, this generates a square wave that encodes rotation. A more advanced version of this uses 2 IR sensors to determine rotation direction. The LSM6DS33 IMU module contains an accelerometer and gyroscope and tracks the rotation and acceleration along the X Y and Z axis. The Arduino UNO is used to read data from the encoders and IMU in real- time and sends them to the raspberry pi for processing. The data from the sensors can be exported in .csv format and can contain rotation, speed, and position data at discrete timesteps. This is a standard architecture for most robots. Real-time sensors such as IMU modules and encoders are controlled by a microcontroller instead of a sin- gle-board computer like the Raspberry Pi. The main advantages are modularity, and ease of development, meaning that the modules can be developed separately and then integrated. Figure 1: Platform architecture 206 4. Localization and positioning model, dataflow charts, and results The localization and positioning model definition in this paper is based on a three-layered model, including an input data layer, processing data layer, and results representation, including calibration of sensors, when appropriate. As was mentioned in the previous section we assume, that there are several different sensors, i.e. IMU, encoders, and optical IR/laser optical sensor. They are providing input data of the current position and location of the service robot. Data acquisition is applied to obtain all data from these sensors, mounted on the appropriate place of the service’s robot body. An Arduino UNO is used to get data from the IMU and encoders. This collected data is sent for processing using the Raspberry PI module, connected to the Arduino UNO module. The data could use raw format or CSV format. For processing, input data could be used Matlab functions, including graph representation of the results. The Kalman Filter algorithm is used to calculate the service robot position on two dimensions – position and orientation estima- tion. The optical velocity estimation, using optical sensors, as mentioned, could use more than one optical sensor, in order to improve service robot accuracy. In regards to such improvement, it will be also beneficial to include in estimation the time variable, in order to achieve proper estimation on the position leveling, based on the motion of the service robot. The model described is graphically represented in Figure 2. Figure 2: Localization model 207 Figure 3: Localization data flow Based on the model mentioned above, and KF algorithm, we present a data flow model for the localization of robots in four steps: • The first step is to obtain data from wheel encoders and pre-defined maps. In this step also a data from Arduino UNO, data from the camera, and optical flow is gathered and presented for measuring • The second step produces a localization prediction of the robot, based on measurement prediction that comes from a predefined map and wheel encoder; • On the third step, a matching between measured information and ob- served information, comes from a camera, optical flow sensor, or any other sensor attached to the robot. In this step, the robot processes both dataflows and the outcome is the best match between both features; • In the estimation step, the KF method fuses the matching information in order to update the robot’s position. As mentioned above, the robot’s localization estimates the robot position from the environment map. The overall localization model and data flow create a more exact environment map from the information on the robot’s accurate path. In this way, the proposed model utilizes the data collected from the different types of robot’s sensors, comparing it with a predefined map data, and predict accurate robot path. Aggregated data are based on the current positioning of the robot, estimated from the surrounding environment, predefined maps, boundaries and sensors. 208 In dynamic environments, if the SLAM method with EKF algorithm is used, it will allow the robot to update existing or create new maps, while it is moving and closely tracking and observing all objects around. The expected results of this method should be visualized graphically and in a table format. In this way it will be easy for operators to calibrate, where possible, the sensors. The final setup must be deployed on the expert system (middleware), which manages service robots. Also, to validate the described localization method, the next step is an experiment to be planned and conducted. In this way, the concept of the localization model could be proved. 5. Conclusions The combination of optical sensors, IMU and encoders in service robots could bring better robustness and accuracy of the service robots’ positioning and localization. All sensors will track and measure the overall velocity, angular ve- locity, orientation, and positioning, as well as the acceleration of the robot. A localization model is based on the Kalman filter algorithm for a mobile robot that utilizes mentioned sensors. Proposed approach covers two of the three prototype development layers of IHE concept: infrastructure layer and data processing and management layer. The third, service layer depends of the application, where the service robots will oper- ate. Considering IHE as an operational environment and reviewing the mitigation or avoiding the uncertainty of position estimation/localization of the service ro- bots by using KF algorithm, the robots are capable to perform tasks and activities in home environment. The proper localization of service robots at any moment of their operation gives opportunity of mounting additional equipment (or sensors, or actuators) on them, with ability to perform secondary tasks. There are lot of examples, including cameras (for security and/or monitoring purposes), small carry-on platforms (moving objects from location A to B), robotic hand with fin- gers to press buttons and/or perform tasks on different height etc. The results of the proposed approach including the KF algorithm and rep- resented model could be validated and verified in several experiments. The vali- dation list shall include key indicators of accuracy of localization and position- ing by detecting the correct robot path and avoiding uncertainty of any skidding caused by different obstacles. Another validation is to key indicators of accuracy of performance and number of sensors depending on the service robot size and its application. 209 6. Acknowledgments This paper is prepared with the support of MIRACle: Mechatronics, Innova- tion, Robotics, Automation, Clean technologies – Establishment and develop- ment of a Center for Competence in Mechatronics and Clean Technologies – Laboratory Intelligent Urban Environment, funded by the Operational Program Science and Education for smart growth 2014–2020, Project BG 05M2OP001- 1.002-0011. 7. References [1] International Federation of Robotics (2015a) Definition of service robots, URL: http://www.ifr.org/service-robots. Accessed 17 Feb 2015. [2] Sprenger, M., Mettler, T. Service Robots. Bus Inf Syst Eng 57, 271–274 (2015). https://doi.org/10.1007/s12599-015-0389-x. [3] Kawamura, Kazuhiko, et al. “Design philosophy for service robots.” Ro- botics and Autonomous Systems 18.1-2 (1996): 109–116. [4] International Organization for Standardization, Robotics — Application of ISO 8373:2021, 2021, URL: https://www.iso.org/obp/ui/#iso:std:75539:en. [5] Prabin Kumar Panigrahi, Sukant Kishoro Bisoy, “Localization strategies for autonomous mobile robots: A review”. URL: https://doi.org/10.1016/j. jksuci.2021.02.015. 210