Trash detection on the floor with IntelRealSense Lesia Mochurad and Bohdan Herasym Lviv Polytechnic National University, 12 Bandera street, Lviv, 79013, Ukraine Abstract In our rapidly advancing world, the convenience of modern living comes hand in hand with a growing concern: waste management. As populations surge and urban centers expand, the accumulation of trash has become an ever-looming challenge. The importance of efficient trash detection and pickup systems cannot be overstated in this context. This paper introduces a novel approach for the automatic detection of trash and debris on indoor floors through the utilization of Intel RealSense technology. The system leverages depth-sensing and computer vision capabilities to identify and classify various types of litter, enabling efficient and autonomous cleaning processes in residential and commercial environments. By combining the power of Intel RealSense with advanced machine learning algorithms, this solution demonstrates promising results in improving cleanliness and hygiene, reducing manual labor, and enhancing overall maintenance efficiency. Keywords 1 Trash detection, computer vision, trash pick up, deps camera, IntelRealSense 1. Introduction Proper waste management is not just about cleanliness; it is about safeguarding the environment, public health, and the overall well-being of communities. Trash detection and pickup play a pivotal role in preserving the natural beauty of our surroundings, ensuring the cleanliness of our streets, and mitigating the adverse effects of pollution. In this age of environmental awareness, these processes are integral to combating climate change, protecting wildlife, and maintaining the delicate balance of our ecosystems [1, 2]. This vital task extends beyond mere aesthetics. Timely and effective trash detection and pickup are fundamental in preventing the spread of diseases, enhancing the quality of air and water, and creating spaces where people can thrive. Moreover, in the era of smart technology, the integration of innovative solutions like sensors, data analytics, and artificial intelligence has revolutionized waste management, making it not only more efficient but also more environmentally friendly [3]. This introduction sets the stage for a deeper exploration into the significance of trash detection and pickup, shedding light on the multifaceted benefits that these processes offer to our communities, environment, and, ultimately, our collective future. In this paper proposes a method for detecting and picking up trash designed for autonomous cleaning robots to operate indoors. However, this innovative approach extends far beyond its initial scope. Its adaptability and potential for wider application suggest a transformative impact on various scenarios, promising a more efficient and comprehensive solution to waste management challenges [4-6]. Based on the literature search conducted on the topic of the study, eleven articles directly related to the research theme have been selected. They are summarized below in Table 1. ProfIT AI 2023: 3rd International Workshop of IT-professionals on Artificial Intelligence (ProfIT AI 2023), November 20–22, 2023, Waterloo, Canada : lesia.i.mochurad@lpnu.ua (L. Mochurad); bohdan.herasym.mknssh.2022@lpnu.ua (B. Herasym) 0000-0002-4957-1512 (L. Mochurad); 0009-0000-5708-6843 (B. Herasym) ©️ 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Objective: The primary objective of this research is to comprehensively investigate and develop strategies aimed at streamlining the cleaning process and elevating the operational efficiency of cleaning robots. This will be achieved through the implementation of advanced techniques such as garbage recognition and the precise identification of pickup points. Table 1 Review of existing work № Title Instrumentation Task Year Reference 1 A vision-based robotic RPN and the Detection with pursue 2017 [7] grasping system using VGG-16 to detect trash deep learning for garbage sorting 2 A Review of Trash Scopus, Overview of modern 2021 [8] Collecting and Cleaning Google scholar solution for trash Robots detection and cleanup using autonomous robots 3 Optimal selective floor CCTV, SORT, Detection of places 2022 [9] cleaning using deep SSD best suited for trash learning algorithms and clean up reconfigurable robot automatization hTetro 4 Design and ROS Automatization and 2023 [10] Implementation of an optimization of trash Artificial Intelligence of pick up from trash can Things-Based Autonomous Mobile Robot System for Cleaning Garbage 5 OATCR: Outdoor Yolov4, Mask- Development of 2021 [11] Autonomous Trash- RCNN, autonomous robot that Collecting Robot Design collect trash Using YOLOv4-Tiny 6 Autonomous Trash ROS, RNN, Development of tiny 2019 [12] Collector Based on Yolov4 autonomous robot Object Detection Using that collect trash Deep Neural Network 7 Deep Learning Based ROS Development and 2022 [13] Robot for Automatically optimization algorithm Picking Up Garbage on for trash detection and the Grass navigation for autonomous robot 8 Cascaded Machine- CNN, SVM, SSD Classification for trash 2018 [14] Learning Technique for detection Debris Classification in and check for ability to Floor-Cleaning Robot pick up trash from floor Application 9 Deep Learning Based DCNN, MMS Detection of defects 2021 [15] Pavement Inspection and trash on road Using Self- Reconfigurable Robot 10. A cascade ensemble- Edge computing, Data preprocessing 2023 [16] learning model for the SVR tasks, such as filling in deployment at the data gaps, anomaly edge: case on missing detection, data IoT data recovery in dimensionality environmental reduction, and others monitoring systems are important in almost all systems for intelligent analysis of tabular data sets 11 An Obstacle-Finding 2D LiDAR, DFS, A parallel algorithm for 2023 [17] Approach for DBSCAN obstacle detection Autonomous Mobile based on two- Robots Using 2D LiDAR dimensional LiDAR Data sensor data has been developed. Research Object: The focal point of this study centers on cleaning robots as a critical component of modern cleaning practices. The research aims to delve into their capabilities, performance, and potential in terms of organizing and maintaining cleanliness within indoor spaces. Research Subject: The core subject of this investigation pertains to the innovative approach of integrating garbage recognition technology and precise pickup point determination. These innovative methods are expected to serve as a catalyst for improving the overall effectiveness and productivity of cleaning robots, thus addressing the ever-growing demand for efficient and automated cleaning solutions in various indoor environments. 2. Methodic and materials Proposed method consisted of a few parts, diagram provided in Figure 1. Figure 1: Proposed algorithm diagram At the inception of the workflow, the system initiates the data collection process, capturing valuable sensor data from IntelRealSense devices. This data serves as a comprehensive representation of the environment, allowing the system to discern its layout and the presence of various objects, with a particular focus on identifying trash items. The subsequent phase of the operation involves intricate algorithms designed to not only detect the presence of trash but also classify it into specific categories and pinpoint its precise location on the floor. This multifaceted approach is pivotal for enabling the system to efficiently and accurately recognize various types of waste materials, an essential aspect of any cleaning and waste management process. Following this, the data is passed on to a specialized component within the system, tasked with the conversion of the pixel-based position representation into SI units. This conversion serves to standardize the location information, making it universally comprehensible and facilitating seamless integration with other devices and software. In the final segment of this complex process, the system utilizes advanced techniques to approximate the shape and attributes of the detected trash items. This analysis is crucial in determining the most suitable points for the robotic gripper to engage with the waste materials. By understanding the intricacies of the trash's structure, the system can make well-informed decisions about the optimal pickup points, ensuring a secure and efficient removal process. This step is instrumental in enhancing the system's overall efficiency and effectiveness in carrying out cleaning and waste management tasks. 2.1 Detection and classification The methodology employed in our proposed system leverages the formidable capabilities of YOLOv5 [18], a state-of-the-art object detection model, which has been trained on the TACO dataset [19]. This robust training process equips the model with a deep understanding of a wide array of object classes, allowing it to proficiently recognize and classify various items within its field of view. However, when it comes to the crucial task of selecting the appropriate objects for pickup, the system needs to discern whether an object is squishable, solid, or unpickable, as these characteristics significantly impact the grasping and handling strategy. To accommodate this decision-making process, all the classes from the TACO dataset are propagated into a smaller set of overarching categories, which primarily include squishable, solid, and unpickable. This strategic classification process is instrumental in ensuring that the robotic system can make informed decisions when it comes to the retrieval of objects, taking into account their physical properties and suitability for pickup. By categorizing objects into these key classes, the system can fine-tune its actions, optimizing its efficiency and effectiveness in handling a diverse range of items in real-world scenarios, making it an invaluable asset in the realm of automation and robotics (see Figure 2). Figure 2: Training process 2.2 Trash region Selection After receiving bounding boxes from our recognition model, we extract and focus on the specific regions of interest within the depth image. This process streamlines subsequent analysis, improves computational efficiency, and ensures our algorithms work effectively by concentrating on the most relevant areas. This targeted approach reduces noise, optimizes resources, and enhances the overall performance of our computer vision system. 2.3 Trash modeling Trash modeling proceeds in two stages, first stage we split all points from selected areas into regions with heights equal to heights of gripper clams. Then for each region we project all points into the bottom of the region, and find minimal bounding box. Rotating calipers is a geometric algorithm that can be used to find the minimum bounding box (also known as the minimum area rectangle) of a set of points in a 2D plane. This algorithm is efficient and works by rotating two perpendicular calipers to envelope the points. Here are the steps to find the minimal bounding box using the rotating calipers algorithm: Find the convex hull of the given set of points. The convex hull is the smallest convex polygon that contains all the input points. Author use Graham Scan algorithm. Initialie two calipers (long and short) that are initially aligned with two edges of the convex hull. The long caliper should be aligned with the longest edge of the convex hull, and the short caliper should be perpendicular to the long caliper. Calculate the area of the rectangle formed by the two calipers. This rectangle represents a potential minimal bounding box. Rotate the short caliper counterclockwise by a small angle (e.g., one degree or a smaller increment) while keeping the long caliper aligned with the longest edge of the convex hull. For each rotation of the short caliper, calculate the area of the rectangle formed by the two calipers and check if it's smaller than the previous minimum area found. If it is, update the minimum area. Continue rotating the short caliper until it completes a full rotation, which is 180 degrees. Make sure to keep the long caliper aligned with the convex hull's longest edge throughout. After completing a full rotation, you will have the minimal bounding box with the minimum area. To find the actual coordinates of the minimal bounding box, you can use the endpoints of the long and short calipers and the orientation of the calipers. The rotating calipers algorithm guarantees finding the minimum area bounding box efficiently with a time complexity of O(n), where n is the number of points in the convex hull. This method is particularly useful when you need to find the tightest bounding box (see Figure 3) for a set of points in 2D space. Figure 3: Bounding box example Then we extend the bounding box back to the selected region height. 2.4 Point Selection Algorithm After meddling we received stacks of cuboids then we went from top to bottom, checking cuboids for selection. If the sides are close enough to be clamped by a gripper we go to the next rectangle. We check two scenarios: if the cuboid's lower part is bound to parallel sides of the top cuboid, we mark it as pickable. If no last cuboid becomes our pickable option. This algorithm is described below: cuboidStack; solution=cuboidStack.top; CheckForBestCuboid(cuboidStack,none) function CheckForBestCuboid(cuboidStack, restrictedArea): top=cuboidStack.pop(); if(ValidCuboid(top,restrictedArea)) solution=top restrictedArea+=RestrictedArea(top) CheckForBestCuboid(cuboidStack, restrictedArea) else: end function ValidCuboid(cuboid,restrictedArea): if(cuboid.sides in restrictedArea) return false else return true function RestrictedArea(cuboid): width= cuboid.width in restrictedArea width= cuboid.length in restrictedArea if(width>GripperClamSize) restrictedArea+= cuboid.AreaOutofWidthSides if(length>GripperClamSize) restrictedArea+= cuboid.AreaOutofLengthSides 3. Numerous experiments Since the algorithm consists of two parts, experiments were initially conducted with the recognition algorithm. When it achieved satisfactory accuracy, experiments were then performed with the second algorithm. Therefore, the recognition algorithm was first investigated using 10% of the data from the TACO dataset. Additionally, a small amount of data collected by the article's authores, which accounts for approximately 1% of the TACO dataset, was used for testing. This approach was chosen because the TACO dataset is not very large. The model performance mainly depends on correct classification, not precision of the bounding box, because the bounding box is good enough to decimate any noise or unwanted objects from the frame. This sequential approach to experimenting and validating the two parts of the algorithm allowed for a more structured and efficient development process, ensuring that the recognition component was working well before moving on to the second part. It helped in identifying potential issues and fine-tuning the algorithm's performance (see Figure 4). a) b) Figure 4: Model examples The second part of the discussion focuses on an algorithm designed to streamline the process of selecting a pickup point, which has been rigorously compared with a more straightforward, naive pickup approach. In the case of the naive pickup method, the primary strategy involves reaching for the object's center point and attempting to grasp it. Now, let's delve into the specifics of these two pickup methods. In the case of the naive pickup, the objective is clear: reach for the center of the object and grab it. However, there are specific criteria that determine whether the pickup is considered successful. If the gripper fails to raise the object and hold it securely for at least three seconds, the attempt is considered unsuccessful. This time threshold of three seconds serves as a crucial benchmark for evaluating the efficacy of the naive pickup strategy. It's important to note that both of these methods have demonstrated commendable performance when it comes to picking up squishable objects. The testing primarily focused on items such as pieces of paper and plastic bags, where both the algorithmic approach and the naive method exhibited a high degree of reliability and efficiency (see Figure 5). However, the stark contrast between these two pickup methods becomes evident when dealing with solid objects. In such cases, the limitations of the naive pickup method become pronounced. One of the significant shortcomings of the naive approach is its failure to consider the orientation of the object, which results in frequent failures. Solid objects often exhibit irregular shapes and varying orientations, making it a challenging task for the naive method to consistently achieve successful pickups. This limitation underscores the importance of the algorithmic approach, which takes into account various factors, including the object's orientation, in order to increase the success rate of pickups, particularly when dealing with solid objects. In summary, while both pickup methods excel at handling squishable objects, the algorithmic approach shines when it comes to solid objects due to its ability to adapt to varying orientations. This contrast highlights the significance of adopting more sophisticated strategies when dealing with complex, non-uniform objects in the realm of pickup and manipulation. Figure 5: Training comparison The automatic detection of waste on the floor using Intel RealSense technology developed in this paper has important environmental, economic, and social implications: 1. Environmental importance: • Recycling efficiency (аutomatic waste detection can promote efficient collection); • Pollution prevention (quick identification). 2. Economic importance: • Resource optimization (efficient waste detection); • Cost savings (by automating waste detection and collection). 3. Social importance: • Improved hygiene (automatic waste detection); • Awareness raising (introduction of waste detection technology can increase public awareness of waste management issues). 4. Conclusions In conclusion, effective trash pickup and detection systems play a pivotal role in addressing the ever-growing global issue of waste management. As our world continues to grapple with the challenges of population growth, urbanization, and environmental degradation, it is imperative that we embrace innovative technologies and strategies to manage our waste more efficiently and sustainably. Trash pickup systems, ranging from traditional municipal services to advanced automated solutions, help keep our communities clean and hygienic, ensuring a healthier and more pleasant living environment. Moreover, by reducing litter and waste in public spaces, they contribute to the preservation of our natural habitats, protecting wildlife and ecosystems from the harmful effects of pollution. The integration of detection technologies, such as sensors and machine learning algorithms, into waste management processes offers a promising avenue for improving efficiency and reducing environmental impact. These systems not only enable early detection of overflowing bins, but they also optimize collection routes, reducing fuel consumption and greenhouse gas emissions. This not only benefits the environment but also lowers operational costs for waste management services. Furthermore, the use of data analytics and real-time monitoring allows for a more data- driven and responsive approach to waste management. This means that resources can be allocated more efficiently, and waste collection can be tailored to the specific needs of each community. Such data-driven systems can even encourage more responsible waste disposal practices among individuals, fostering a culture of sustainability and environmental awareness. In summary, the convergence of traditional trash pickup services with cutting-edge detection technologies represents a significant step forward in our ongoing efforts to create cleaner, more sustainable, and environmentally conscious communities. The collective action of governments, businesses, and individuals, in tandem with the advancement of these technologies, will undoubtedly contribute to a brighter and more sustainable future. By harnessing the power of innovation and data-driven decision-making, we can take meaningful strides towards addressing the challenges of waste management in the 21st century and leaving a cleaner, healthier planet for generations to come. Acknowledgements The authors would like to thank the Armed Forces of Ukraine for providing security to perform this work. This work has become possible only because of the resilience and courage of the Ukrainian Army. References [1] Dr. Rafique Anjum, Bilal Ahmed Wani, The Concept of Ecological Balance and Environmental conservation: An Islamic perspective. Suraj Punj Journal For Multidisciplinary Research, (2020), vol. 8, no. 12, pp. 45-58. [2] V. Kovtun, I. Izonin, and M. Gregus, Model of functioning of the centralized wireless information ecosystem focused on multimedia streaming, Egyptian Informatics Journal, vol. 23, no. 4. Elsevier BV, pp. 89–96, Dec. (2022). doi: 10.1016/j.eij.2022.06.009. [3] D. Chumachenko, P. Piletskiy, M. Sukhorukova, T. Chumachenko, Predictive Model of Lyme Disease Epidemic Process Using Machine Learning Approach, Applied Sciences (Switzerland), 12 (9), art. no. 4282,(2022), doi: 10.3390/app12094282. [4] V. Semkovych, V. Shymanskyi, Combining OCR Methods to Improve Handwritten Text Recognition with Low System Technical Requirements. In: Hu, Z., Wang, Y., He, M. (eds) Advances in Intelligent Systems, Computer Science and Digital Economics IV. CSDEIS 2022. Lecture Notes on Data Engineering and Communications Technologies, (2023), vol 158. Springer, Cham. doi:10.1007/978-3-031-24475-9_56. [5] Jamil Abedalrahim Jamil Alsayaydeh, Azwan Aziz, A. I. A. Rahman, Syed Najib Syed Salim, Maslan Zainon, Zikri Abadi Baharudin, Muhammad Inam Abbasi and Adam Wong Yoon Khang, Development of programmable home security using GSM system for early prevention, ARPN Journal of Engineering and Applied Sciences, (2021), 16(1):88-97. [6] W.A. Indra et. al., Development of Security System Using Motion Sensor Powered by RF Energy Harvesting. 2020 IEEE Student Conference on Research and Development, (2020), SCOReD 2020 9250984, pp. 254-258. [7] C. Zhihong, Z. Hebin, W. Yanbo, L. Binyan and L. Yu, A vision-based robotic grasping system using deep learning for garbage sorting, 2017 36th Chinese Control Conference (CCC), Dalian, China, (2017), pp. 11223-11226, doi: 10.23919/ChiCC.2017.8029147. [8] S. S. Chandra, M. Kulshreshtha, P. Randhawa, A Review of Trash Collecting and Cleaning Robots, 2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India, (2021), pp. 1-5, doi: 10.1109/ICRITO51393.2021.9596551. [9] B. Ramalingam et al., Optimal selective floor cleaning using deep learning algorithms and reconfigurable robot hTetro, (2022), Sci Rep 12, 15938. doi.org/10.1038/s41598-022-19249-7. [10] L.-B. Chen, X. -R. Huang, et. al., Design and Implementation of an Artificial Intelligence of Things-Based Autonomous Mobile Robot System for Cleaning Garbage, in IEEE Sensors Journal, (2023), vol. 23, no. 8, pp. 8909-8922. doi: 10.1109/JSEN.2023.3254902. [11] Ejour Kulshreshtha, Medhasvi Chandra, et. al. Outdoor Autonomous Trash-Collecting Robot Design Using YOLOv4-Tiny Electronics, (2021), 10(18):2079-9292. [12] S. Hossain, B. Debnath, et. al. Autonomous Trash Collector Based on Object Detection Using Deep Neural Network, TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON), Kochi, India, (2019), pp. 1406-1410, doi: 10.1109/TENCON.2019.8929270. [13] J. Bai, S. Lian, Z. Liu, K. Wang and D. Liu, Deep Learning Based Robot for Automatically Picking Up Garbage on the Grass, in IEEE Transactions on Consumer Electronics, (2018), vol. 64, no. 3, pp. 382-389, doi: 10.1109/TCE.2018.2859629. [14] B. Ramalingam, A.K. Lakshmanan, et. al. Cascaded Machine-Learning Technique for Debris Classification in Floor-Cleaning Robot Application. Appl. Sci., (2018), 8, 2649. doi:10.3390/app8122649. [15] B. Ramalingam, A.A. Hayat, et.al. Deep Learning Based Pavement Inspection Using Self- Reconfigurable Robot. Sensors, (2021), 21, 2595. doi:10.3390/s21082595. [16] I. Izonin, R. Tkachenko, I. Krak, O. Berezsky, I. Shevchuk, S.K., A cascade ensemble-learning model for the deployment at the edge: case on missing IoT data recovery in environmental monitoring systems. Front. Environ. Sci., (2023), 11:1295526. doi: 10.3389/fenvs.2023.1295526. [17] L. Mochurad, Y. Hladun, R. Tkachenko, An Obstacle-Finding Approach for Autonomous Mobile Robots Using 2D LiDAR Data. Big Data and Cognitive Computing. (2023), 7(1):43. doi:10.3390/bdcc7010043. [18] J. Zhang, Z. Chen, et. al., Faster and Lightweight: An Improved YOLOv5 Object Detector for Remote Sensing Images. Remote Sensing, (2023), 15(20):4974. doi.org/10.3390/rs15204974. [19] TACO dataset. Available online: http://tacodataset.org/explore (accessed on 3 November 2023).