In our rapidly advancing world, the convenience of modern living comes hand in hand with a growing concern: waste management. As populations surge and urban centers expand, the accumulation of trash has become an ever-looming challenge. The importance of efficient trash detection and pickup systems cannot be overstated in this context. This paper introduces a novel approach for the automatic detection of trash and debris on indoor floors through the utilization of Intel RealSense technology. The system leverages depth-sensing and computer vision capabilities to identify and classify various types of litter, enabling efficient and autonomous cleaning processes in residential and commercial environments. By combining the power of Intel RealSense with advanced machine learning algorithms, this solution demonstrates promising results in improving cleanliness and hygiene, reducing manual labor, and enhancing overall maintenance efficiency.
Proper waste management is not just about cleanliness; it is about safeguarding the
environment, public health, and the overall well-being of communities. Trash detection and
pickup play a pivotal role in preserving the natural beauty of our surroundings, ensuring the
cleanliness of our streets, and mitigating the adverse effects of pollution. In this age of
environmental awareness, these processes are integral to combating climate change, protecting
wildlife, and maintaining the delicate balance of our ecosystems [
This vital task extends beyond mere aesthetics. Timely and effective trash detection and
pickup are fundamental in preventing the spread of diseases, enhancing the quality of air and
water, and creating spaces where people can thrive. Moreover, in the era of smart technology,
the integration of innovative solutions like sensors, data analytics, and artificial intelligence has
revolutionized waste management, making it not only more efficient but also more
environmentally friendly [
This introduction sets the stage for a deeper exploration into the significance of trash detection and pickup, shedding light on the multifaceted benefits that these processes offer to our communities, environment, and, ultimately, our collective future.
In this paper proposes a method for detecting and picking up trash designed for autonomous
cleaning robots to operate indoors. However, this innovative approach extends far beyond its
initial scope. Its adaptability and potential for wider application suggest a transformative
impact on various scenarios, promising a more efficient and comprehensive solution to waste
management challenges [
Based on the literature search conducted on the topic of the study, eleven articles directly related to the research theme have been selected. They are summarized below in Table 1.
Objective: The primary objective of this research is to comprehensively investigate and develop strategies aimed at streamlining the cleaning process and elevating the operational efficiency of cleaning robots. This will be achieved through the implementation of advanced techniques such as garbage recognition and the precise identification of pickup points.
solution for trash detection and cleanup using autonomous robots
optimization algorithm for trash detection and navigation for autonomous robot
Research Object: The focal point of this study centers on cleaning robots as a critical component of modern cleaning practices. The research aims to delve into their capabilities, performance, and potential in terms of organizing and maintaining cleanliness within indoor spaces.
Research Subject: The core subject of this investigation pertains to the innovative approach of integrating garbage recognition technology and precise pickup point determination. These innovative methods are expected to serve as a catalyst for improving the overall effectiveness and productivity of cleaning robots, thus addressing the ever-growing demand for efficient and automated cleaning solutions in various indoor environments.
Proposed method consisted of a few parts, diagram provided in Figure 1.
At the inception of the workflow, the system initiates the data collection process, capturing valuable sensor data from IntelRealSense devices. This data serves as a comprehensive representation of the environment, allowing the system to discern its layout and the presence of various objects, with a particular focus on identifying trash items.
The subsequent phase of the operation involves intricate algorithms designed to not only detect the presence of trash but also classify it into specific categories and pinpoint its precise location on the floor. This multifaceted approach is pivotal for enabling the system to efficiently and accurately recognize various types of waste materials, an essential aspect of any cleaning and waste management process.
Following this, the data is passed on to a specialized component within the system, tasked with the conversion of the pixel-based position representation into SI units. This conversion serves to standardize the location information, making it universally comprehensible and facilitating seamless integration with other devices and software.
In the final segment of this complex process, the system utilizes advanced techniques to approximate the shape and attributes of the detected trash items. This analysis is crucial in determining the most suitable points for the robotic gripper to engage with the waste materials. By understanding the intricacies of the trash's structure, the system can make well-informed decisions about the optimal pickup points, ensuring a secure and efficient removal process. This step is instrumental in enhancing the system's overall efficiency and effectiveness in carrying out cleaning and waste management tasks.
The methodology employed in our proposed system leverages the formidable capabilities of YOLOv5 [18], a state-of-the-art object detection model, which has been trained on the TACO dataset [19]. This robust training process equips the model with a deep understanding of a wide array of object classes, allowing it to proficiently recognize and classify various items within its field of view.
However, when it comes to the crucial task of selecting the appropriate objects for pickup, the system needs to discern whether an object is squishable, solid, or unpickable, as these characteristics significantly impact the grasping and handling strategy. To accommodate this decision-making process, all the classes from the TACO dataset are propagated into a smaller set of overarching categories, which primarily include squishable, solid, and unpickable.
This strategic classification process is instrumental in ensuring that the robotic system can make informed decisions when it comes to the retrieval of objects, taking into account their physical properties and suitability for pickup. By categorizing objects into these key classes, the system can fine-tune its actions, optimizing its efficiency and effectiveness in handling a diverse range of items in real-world scenarios, making it an invaluable asset in the realm of automation and robotics (see Figure 2).
After receiving bounding boxes from our recognition model, we extract and focus on the specific regions of interest within the depth image. This process streamlines subsequent analysis, improves computational efficiency, and ensures our algorithms work effectively by concentrating on the most relevant areas. This targeted approach reduces noise, optimizes resources, and enhances the overall performance of our computer vision system.
Trash modeling proceeds in two stages, first stage we split all points from selected areas into regions with heights equal to heights of gripper clams.
Then for each region we project all points into the bottom of the region, and find minimal bounding box.
Rotating calipers is a geometric algorithm that can be used to find the minimum bounding box (also known as the minimum area rectangle) of a set of points in a 2D plane. This algorithm is efficient and works by rotating two perpendicular calipers to envelope the points. Here are the steps to find the minimal bounding box using the rotating calipers algorithm:
Find the convex hull of the given set of points. The convex hull is the smallest convex polygon that contains all the input points. Author use Graham Scan algorithm.
Initialie two calipers (long and short) that are initially aligned with two edges of the convex hull. The long caliper should be aligned with the longest edge of the convex hull, and the short caliper should be perpendicular to the long caliper.
Calculate the area of the rectangle formed by the two calipers. This rectangle represents a potential minimal bounding box.
Rotate the short caliper counterclockwise by a small angle (e.g., one degree or a smaller increment) while keeping the long caliper aligned with the longest edge of the convex hull.
For each rotation of the short caliper, calculate the area of the rectangle formed by the two calipers and check if it's smaller than the previous minimum area found. If it is, update the minimum area.
Continue rotating the short caliper until it completes a full rotation, which is 180 degrees. Make sure to keep the long caliper aligned with the convex hull's longest edge throughout.
After completing a full rotation, you will have the minimal bounding box with the minimum area.
To find the actual coordinates of the minimal bounding box, you can use the endpoints of the long and short calipers and the orientation of the calipers.
The rotating calipers algorithm guarantees finding the minimum area bounding box efficiently with a time complexity of O(n), where n is the number of points in the convex hull. This method is particularly useful when you need to find the tightest bounding box (see Figure 3) for a set of points in 2D space.
Then we extend the bounding box back to the selected region height.
After meddling we received stacks of cuboids then we went from top to bottom, checking cuboids for selection.
If the sides are close enough to be clamped by a gripper we go to the next rectangle.
We check two scenarios: if the cuboid's lower part is bound to parallel sides of the top cuboid, we mark it as pickable.
If no last cuboid becomes our pickable option. This algorithm is described below: cuboidStack; solution=cuboidStack.top; CheckForBestCuboid(cuboidStack,none) function CheckForBestCuboid(cuboidStack, restrictedArea): top=cuboidStack.pop(); if(ValidCuboid(top,restrictedArea)) solution=top restrictedArea+=RestrictedArea(top)
CheckForBestCuboid(cuboidStack, restrictedArea) else:
end function ValidCuboid(cuboid,restrictedArea): if(cuboid.sides in restrictedArea)
return false else
return true function RestrictedArea(cuboid): width= cuboid.width in restrictedArea width= cuboid.length in restrictedArea if(width>GripperClamSize)
restrictedArea+= cuboid.AreaOutofWidthSides if(length>GripperClamSize)
restrictedArea+= cuboid.AreaOutofLengthSides
Since the algorithm consists of two parts, experiments were initially conducted with the recognition algorithm. When it achieved satisfactory accuracy, experiments were then performed with the second algorithm. Therefore, the recognition algorithm was first investigated using 10% of the data from the TACO dataset. Additionally, a small amount of data collected by the article's authores, which accounts for approximately 1% of the TACO dataset, was used for testing. This approach was chosen because the TACO dataset is not very large.
The model performance mainly depends on correct classification, not precision of the bounding box, because the bounding box is good enough to decimate any noise or unwanted objects from the frame.
This sequential approach to experimenting and validating the two parts of the algorithm allowed for a more structured and efficient development process, ensuring that the recognition component was working well before moving on to the second part. It helped in identifying potential issues and fine-tuning the algorithm's performance (see Figure 4).
a) b)
The second part of the discussion focuses on an algorithm designed to streamline the process of selecting a pickup point, which has been rigorously compared with a more straightforward, naive pickup approach. In the case of the naive pickup method, the primary strategy involves reaching for the object's center point and attempting to grasp it.
Now, let's delve into the specifics of these two pickup methods. In the case of the naive pickup, the objective is clear: reach for the center of the object and grab it. However, there are specific criteria that determine whether the pickup is considered successful. If the gripper fails to raise the object and hold it securely for at least three seconds, the attempt is considered unsuccessful. This time threshold of three seconds serves as a crucial benchmark for evaluating the efficacy of the naive pickup strategy.
It's important to note that both of these methods have demonstrated commendable performance when it comes to picking up squishable objects. The testing primarily focused on items such as pieces of paper and plastic bags, where both the algorithmic approach and the naive method exhibited a high degree of reliability and efficiency (see Figure 5).
However, the stark contrast between these two pickup methods becomes evident when dealing with solid objects. In such cases, the limitations of the naive pickup method become pronounced. One of the significant shortcomings of the naive approach is its failure to consider the orientation of the object, which results in frequent failures. Solid objects often exhibit irregular shapes and varying orientations, making it a challenging task for the naive method to consistently achieve successful pickups. This limitation underscores the importance of the algorithmic approach, which takes into account various factors, including the object's orientation, in order to increase the success rate of pickups, particularly when dealing with solid objects.
In summary, while both pickup methods excel at handling squishable objects, the algorithmic approach shines when it comes to solid objects due to its ability to adapt to varying orientations. This contrast highlights the significance of adopting more sophisticated strategies when dealing with complex, non-uniform objects in the realm of pickup and manipulation.
The automatic detection of waste on the floor using Intel RealSense technology developed in this paper has important environmental, economic, and social implications: 1. Environmental importance: • Recycling efficiency (аutomatic waste detection can promote efficient collection); • Pollution prevention (quick identification). 2. Economic importance: • Resource optimization (efficient waste detection); • Cost savings (by automating waste detection and collection). 3. Social importance: • Improved hygiene (automatic waste detection); • Awareness raising (introduction of waste detection technology can increase public awareness of waste management issues).
In conclusion, effective trash pickup and detection systems play a pivotal role in addressing the ever-growing global issue of waste management. As our world continues to grapple with the challenges of population growth, urbanization, and environmental degradation, it is imperative that we embrace innovative technologies and strategies to manage our waste more efficiently and sustainably.
Trash pickup systems, ranging from traditional municipal services to advanced automated solutions, help keep our communities clean and hygienic, ensuring a healthier and more pleasant living environment. Moreover, by reducing litter and waste in public spaces, they contribute to the preservation of our natural habitats, protecting wildlife and ecosystems from the harmful effects of pollution.
The integration of detection technologies, such as sensors and machine learning algorithms, into waste management processes offers a promising avenue for improving efficiency and reducing environmental impact. These systems not only enable early detection of overflowing bins, but they also optimize collection routes, reducing fuel consumption and greenhouse gas emissions. This not only benefits the environment but also lowers operational costs for waste management services.
Furthermore, the use of data analytics and real-time monitoring allows for a more datadriven and responsive approach to waste management. This means that resources can be allocated more efficiently, and waste collection can be tailored to the specific needs of each community. Such data-driven systems can even encourage more responsible waste disposal practices among individuals, fostering a culture of sustainability and environmental awareness.
In summary, the convergence of traditional trash pickup services with cutting-edge detection technologies represents a significant step forward in our ongoing efforts to create cleaner, more sustainable, and environmentally conscious communities. The collective action of governments, businesses, and individuals, in tandem with the advancement of these technologies, will undoubtedly contribute to a brighter and more sustainable future. By harnessing the power of innovation and data-driven decision-making, we can take meaningful strides towards addressing the challenges of waste management in the 21st century and leaving a cleaner, healthier planet for generations to come.
The authors would like to thank the Armed Forces of Ukraine for providing security to perform this work. This work has become possible only because of the resilience and courage of the Ukrainian Army. [11] Ejour Kulshreshtha, Medhasvi Chandra, et. al. Outdoor Autonomous Trash-Collecting Robot
Design Using YOLOv4-Tiny Electronics, (2021), 10(18):2079-9292. [12] S. Hossain, B. Debnath, et. al. Autonomous Trash Collector Based on Object Detection Using Deep Neural Network, TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON), Kochi, India, (2019), pp. 1406-1410, doi: 10.1109/TENCON.2019.8929270. [13] J. Bai, S. Lian, Z. Liu, K. Wang and D. Liu, Deep Learning Based Robot for Automatically Picking Up Garbage on the Grass, in IEEE Transactions on Consumer Electronics, (2018), vol. 64, no. 3, pp. 382-389, doi: 10.1109/TCE.2018.2859629. [14] B. Ramalingam, A.K. Lakshmanan, et. al. Cascaded Machine-Learning Technique for Debris Classification in Floor-Cleaning Robot Application. Appl. Sci., (2018), 8, 2649. doi:10.3390/app8122649. [15] B. Ramalingam, A.A. Hayat, et.al. Deep Learning Based Pavement Inspection Using Self
Reconfigurable Robot. Sensors, (2021), 21, 2595. doi:10.3390/s21082595. [16] I. Izonin, R. Tkachenko, I. Krak, O. Berezsky, I. Shevchuk, S.K., A cascade ensemble-learning model for the deployment at the edge: case on missing IoT data recovery in environmental monitoring systems. Front. Environ. Sci., (2023), 11:1295526. doi: 10.3389/fenvs.2023.1295526. [17] L. Mochurad, Y. Hladun, R. Tkachenko, An Obstacle-Finding Approach for Autonomous Mobile Robots Using 2D LiDAR Data. Big Data and Cognitive Computing. (2023), 7(1):43. doi:10.3390/bdcc7010043. [18] J. Zhang, Z. Chen, et. al., Faster and Lightweight: An Improved YOLOv5 Object Detector for
R e m o t e S e n s i n g I m a g e s . R e m o t e S e n s i n g , ( 2 0 2 3 ) , 1 5 ( 2 0 ) : 4 9 7 4 . doi.org/10.3390/rs15204974. [19] TACO dataset. Available online: http://tacodataset.org/explore (accessed on 3 November 2023).