=Paper=
{{Paper
|id=Vol-3641/paper10
|storemode=property
|title=Trash detection on the floor with IntelRealSense
|pdfUrl=https://ceur-ws.org/Vol-3641/paper10.pdf
|volume=Vol-3641
|authors=Lesia Mochurad,Bohdan Herasym
|dblpUrl=https://dblp.org/rec/conf/profitai/MochuradH23
}}
==Trash detection on the floor with IntelRealSense==
https://ceur-ws.org/Vol-3641/paper10.pdf
Trash detection on the floor with IntelRealSense
Lesia Mochurad and Bohdan Herasym
Lviv Polytechnic National University, 12 Bandera street, Lviv, 79013, Ukraine
Abstract
In our rapidly advancing world, the convenience of modern living comes hand in hand with a growing
concern: waste management. As populations surge and urban centers expand, the accumulation of
trash has become an ever-looming challenge. The importance of efficient trash detection and pickup
systems cannot be overstated in this context. This paper introduces a novel approach for the automatic
detection of trash and debris on indoor floors through the utilization of Intel RealSense technology.
The system leverages depth-sensing and computer vision capabilities to identify and classify various
types of litter, enabling efficient and autonomous cleaning processes in residential and commercial
environments. By combining the power of Intel RealSense with advanced machine learning algorithms,
this solution demonstrates promising results in improving cleanliness and hygiene, reducing manual
labor, and enhancing overall maintenance efficiency.
Keywords 1
Trash detection, computer vision, trash pick up, deps camera, IntelRealSense
1. Introduction
Proper waste management is not just about cleanliness; it is about safeguarding the
environment, public health, and the overall well-being of communities. Trash detection and
pickup play a pivotal role in preserving the natural beauty of our surroundings, ensuring the
cleanliness of our streets, and mitigating the adverse effects of pollution. In this age of
environmental awareness, these processes are integral to combating climate change, protecting
wildlife, and maintaining the delicate balance of our ecosystems [1, 2].
This vital task extends beyond mere aesthetics. Timely and effective trash detection and
pickup are fundamental in preventing the spread of diseases, enhancing the quality of air and
water, and creating spaces where people can thrive. Moreover, in the era of smart technology,
the integration of innovative solutions like sensors, data analytics, and artificial intelligence has
revolutionized waste management, making it not only more efficient but also more
environmentally friendly [3].
This introduction sets the stage for a deeper exploration into the significance of trash
detection and pickup, shedding light on the multifaceted benefits that these processes offer to
our communities, environment, and, ultimately, our collective future.
In this paper proposes a method for detecting and picking up trash designed for autonomous
cleaning robots to operate indoors. However, this innovative approach extends far beyond its
initial scope. Its adaptability and potential for wider application suggest a transformative
impact on various scenarios, promising a more efficient and comprehensive solution to waste
management challenges [4-6].
Based on the literature search conducted on the topic of the study, eleven articles directly
related to the research theme have been selected. They are summarized below in Table 1.
ProfIT AI 2023: 3rd International Workshop of IT-professionals on Artificial Intelligence (ProfIT AI 2023), November
20–22, 2023, Waterloo, Canada
: lesia.i.mochurad@lpnu.ua (L. Mochurad); bohdan.herasym.mknssh.2022@lpnu.ua (B. Herasym)
0000-0002-4957-1512 (L. Mochurad); 0009-0000-5708-6843 (B. Herasym)
©️ 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Objective: The primary objective of this research is to comprehensively investigate and
develop strategies aimed at streamlining the cleaning process and elevating the operational
efficiency of cleaning robots. This will be achieved through the implementation of advanced
techniques such as garbage recognition and the precise identification of pickup points.
Table 1
Review of existing work
№ Title Instrumentation Task Year Reference
1 A vision-based robotic RPN and the Detection with pursue 2017 [7]
grasping system using VGG-16 to detect trash
deep learning for
garbage sorting
2 A Review of Trash Scopus, Overview of modern 2021 [8]
Collecting and Cleaning Google scholar solution for trash
Robots detection and cleanup
using autonomous
robots
3 Optimal selective floor CCTV, SORT, Detection of places 2022 [9]
cleaning using deep SSD best suited for trash
learning algorithms and clean up
reconfigurable robot automatization
hTetro
4 Design and ROS Automatization and 2023 [10]
Implementation of an optimization of trash
Artificial Intelligence of pick up from trash can
Things-Based
Autonomous Mobile
Robot System for
Cleaning Garbage
5 OATCR: Outdoor Yolov4, Mask- Development of 2021 [11]
Autonomous Trash- RCNN, autonomous robot that
Collecting Robot Design collect trash
Using YOLOv4-Tiny
6 Autonomous Trash ROS, RNN, Development of tiny 2019 [12]
Collector Based on Yolov4 autonomous robot
Object Detection Using that collect trash
Deep Neural Network
7 Deep Learning Based ROS Development and 2022 [13]
Robot for Automatically optimization algorithm
Picking Up Garbage on for trash detection and
the Grass navigation for
autonomous robot
8 Cascaded Machine- CNN, SVM, SSD Classification for trash 2018 [14]
Learning Technique for detection
Debris Classification in and check for ability to
Floor-Cleaning Robot pick up trash from floor
Application
9 Deep Learning Based DCNN, MMS Detection of defects 2021 [15]
Pavement Inspection and trash on road
Using Self-
Reconfigurable Robot
10. A cascade ensemble- Edge computing, Data preprocessing 2023 [16]
learning model for the SVR tasks, such as filling in
deployment at the data gaps, anomaly
edge: case on missing detection, data
IoT data recovery in dimensionality
environmental reduction, and others
monitoring systems are important in
almost all systems for
intelligent analysis of
tabular data sets
11 An Obstacle-Finding 2D LiDAR, DFS, A parallel algorithm for 2023 [17]
Approach for DBSCAN obstacle detection
Autonomous Mobile based on two-
Robots Using 2D LiDAR dimensional LiDAR
Data sensor data has been
developed.
Research Object: The focal point of this study centers on cleaning robots as a critical
component of modern cleaning practices. The research aims to delve into their capabilities,
performance, and potential in terms of organizing and maintaining cleanliness within indoor
spaces.
Research Subject: The core subject of this investigation pertains to the innovative approach
of integrating garbage recognition technology and precise pickup point determination. These
innovative methods are expected to serve as a catalyst for improving the overall effectiveness
and productivity of cleaning robots, thus addressing the ever-growing demand for efficient and
automated cleaning solutions in various indoor environments.
2. Methodic and materials
Proposed method consisted of a few parts, diagram provided in Figure 1.
Figure 1: Proposed algorithm diagram
At the inception of the workflow, the system initiates the data collection process, capturing
valuable sensor data from IntelRealSense devices. This data serves as a comprehensive
representation of the environment, allowing the system to discern its layout and the presence of
various objects, with a particular focus on identifying trash items.
The subsequent phase of the operation involves intricate algorithms designed to not only
detect the presence of trash but also classify it into specific categories and pinpoint its precise
location on the floor. This multifaceted approach is pivotal for enabling the system to efficiently
and accurately recognize various types of waste materials, an essential aspect of any cleaning
and waste management process.
Following this, the data is passed on to a specialized component within the system, tasked
with the conversion of the pixel-based position representation into SI units. This conversion
serves to standardize the location information, making it universally comprehensible and
facilitating seamless integration with other devices and software.
In the final segment of this complex process, the system utilizes advanced techniques to
approximate the shape and attributes of the detected trash items. This analysis is crucial in
determining the most suitable points for the robotic gripper to engage with the waste materials.
By understanding the intricacies of the trash's structure, the system can make well-informed
decisions about the optimal pickup points, ensuring a secure and efficient removal process. This
step is instrumental in enhancing the system's overall efficiency and effectiveness in carrying
out cleaning and waste management tasks.
2.1 Detection and classification
The methodology employed in our proposed system leverages the formidable capabilities of
YOLOv5 [18], a state-of-the-art object detection model, which has been trained on the TACO
dataset [19]. This robust training process equips the model with a deep understanding of a wide
array of object classes, allowing it to proficiently recognize and classify various items within its
field of view.
However, when it comes to the crucial task of selecting the appropriate objects for pickup,
the system needs to discern whether an object is squishable, solid, or unpickable, as these
characteristics significantly impact the grasping and handling strategy. To accommodate this
decision-making process, all the classes from the TACO dataset are propagated into a smaller set
of overarching categories, which primarily include squishable, solid, and unpickable.
This strategic classification process is instrumental in ensuring that the robotic system can
make informed decisions when it comes to the retrieval of objects, taking into account their
physical properties and suitability for pickup. By categorizing objects into these key classes, the
system can fine-tune its actions, optimizing its efficiency and effectiveness in handling a diverse
range of items in real-world scenarios, making it an invaluable asset in the realm of automation
and robotics (see Figure 2).
Figure 2: Training process
2.2 Trash region Selection
After receiving bounding boxes from our recognition model, we extract and focus on the
specific regions of interest within the depth image. This process streamlines subsequent
analysis, improves computational efficiency, and ensures our algorithms work effectively by
concentrating on the most relevant areas. This targeted approach reduces noise, optimizes
resources, and enhances the overall performance of our computer vision system.
2.3 Trash modeling
Trash modeling proceeds in two stages, first stage we split all points from selected areas into
regions with heights equal to heights of gripper clams.
Then for each region we project all points into the bottom of the region, and find minimal
bounding box.
Rotating calipers is a geometric algorithm that can be used to find the minimum bounding
box (also known as the minimum area rectangle) of a set of points in a 2D plane. This algorithm
is efficient and works by rotating two perpendicular calipers to envelope the points. Here are
the steps to find the minimal bounding box using the rotating calipers algorithm:
Find the convex hull of the given set of points. The convex hull is the smallest convex polygon
that contains all the input points. Author use Graham Scan algorithm.
Initialie two calipers (long and short) that are initially aligned with two edges of the convex
hull. The long caliper should be aligned with the longest edge of the convex hull, and the short
caliper should be perpendicular to the long caliper.
Calculate the area of the rectangle formed by the two calipers. This rectangle represents a
potential minimal bounding box.
Rotate the short caliper counterclockwise by a small angle (e.g., one degree or a smaller
increment) while keeping the long caliper aligned with the longest edge of the convex hull.
For each rotation of the short caliper, calculate the area of the rectangle formed by the two
calipers and check if it's smaller than the previous minimum area found. If it is, update the
minimum area.
Continue rotating the short caliper until it completes a full rotation, which is 180 degrees.
Make sure to keep the long caliper aligned with the convex hull's longest edge throughout.
After completing a full rotation, you will have the minimal bounding box with the minimum
area.
To find the actual coordinates of the minimal bounding box, you can use the endpoints of the
long and short calipers and the orientation of the calipers.
The rotating calipers algorithm guarantees finding the minimum area bounding box
efficiently with a time complexity of O(n), where n is the number of points in the convex hull.
This method is particularly useful when you need to find the tightest bounding box (see Figure
3) for a set of points in 2D space.
Figure 3: Bounding box example
Then we extend the bounding box back to the selected region height.
2.4 Point Selection Algorithm
After meddling we received stacks of cuboids then we went from top to bottom, checking
cuboids for selection.
If the sides are close enough to be clamped by a gripper we go to the next rectangle.
We check two scenarios: if the cuboid's lower part is bound to parallel sides of the top
cuboid, we mark it as pickable.
If no last cuboid becomes our pickable option. This algorithm is described below:
cuboidStack;
solution=cuboidStack.top;
CheckForBestCuboid(cuboidStack,none)
function CheckForBestCuboid(cuboidStack, restrictedArea):
top=cuboidStack.pop();
if(ValidCuboid(top,restrictedArea))
solution=top
restrictedArea+=RestrictedArea(top)
CheckForBestCuboid(cuboidStack, restrictedArea)
else:
end
function ValidCuboid(cuboid,restrictedArea):
if(cuboid.sides in restrictedArea)
return false
else
return true
function RestrictedArea(cuboid):
width= cuboid.width in restrictedArea
width= cuboid.length in restrictedArea
if(width>GripperClamSize)
restrictedArea+= cuboid.AreaOutofWidthSides
if(length>GripperClamSize)
restrictedArea+= cuboid.AreaOutofLengthSides
3. Numerous experiments
Since the algorithm consists of two parts, experiments were initially conducted with the
recognition algorithm. When it achieved satisfactory accuracy, experiments were then
performed with the second algorithm. Therefore, the recognition algorithm was first
investigated using 10% of the data from the TACO dataset. Additionally, a small amount of data
collected by the article's authores, which accounts for approximately 1% of the TACO dataset,
was used for testing. This approach was chosen because the TACO dataset is not very large.
The model performance mainly depends on correct classification, not precision of the
bounding box, because the bounding box is good enough to decimate any noise or unwanted
objects from the frame.
This sequential approach to experimenting and validating the two parts of the algorithm
allowed for a more structured and efficient development process, ensuring that the recognition
component was working well before moving on to the second part. It helped in identifying
potential issues and fine-tuning the algorithm's performance (see Figure 4).
a) b)
Figure 4: Model examples
The second part of the discussion focuses on an algorithm designed to streamline the process
of selecting a pickup point, which has been rigorously compared with a more straightforward,
naive pickup approach. In the case of the naive pickup method, the primary strategy involves
reaching for the object's center point and attempting to grasp it.
Now, let's delve into the specifics of these two pickup methods. In the case of the naive
pickup, the objective is clear: reach for the center of the object and grab it. However, there are
specific criteria that determine whether the pickup is considered successful. If the gripper fails
to raise the object and hold it securely for at least three seconds, the attempt is considered
unsuccessful. This time threshold of three seconds serves as a crucial benchmark for evaluating
the efficacy of the naive pickup strategy.
It's important to note that both of these methods have demonstrated commendable
performance when it comes to picking up squishable objects. The testing primarily focused on
items such as pieces of paper and plastic bags, where both the algorithmic approach and the
naive method exhibited a high degree of reliability and efficiency (see Figure 5).
However, the stark contrast between these two pickup methods becomes evident when
dealing with solid objects. In such cases, the limitations of the naive pickup method become
pronounced. One of the significant shortcomings of the naive approach is its failure to consider
the orientation of the object, which results in frequent failures. Solid objects often exhibit
irregular shapes and varying orientations, making it a challenging task for the naive method to
consistently achieve successful pickups. This limitation underscores the importance of the
algorithmic approach, which takes into account various factors, including the object's
orientation, in order to increase the success rate of pickups, particularly when dealing with solid
objects.
In summary, while both pickup methods excel at handling squishable objects, the algorithmic
approach shines when it comes to solid objects due to its ability to adapt to varying
orientations. This contrast highlights the significance of adopting more sophisticated strategies
when dealing with complex, non-uniform objects in the realm of pickup and manipulation.
Figure 5: Training comparison
The automatic detection of waste on the floor using Intel RealSense technology developed in
this paper has important environmental, economic, and social implications:
1. Environmental importance:
• Recycling efficiency (аutomatic waste detection can promote efficient collection);
• Pollution prevention (quick identification).
2. Economic importance:
• Resource optimization (efficient waste detection);
• Cost savings (by automating waste detection and collection).
3. Social importance:
• Improved hygiene (automatic waste detection);
• Awareness raising (introduction of waste detection technology can increase public
awareness of waste management issues).
4. Conclusions
In conclusion, effective trash pickup and detection systems play a pivotal role in addressing the
ever-growing global issue of waste management. As our world continues to grapple with the
challenges of population growth, urbanization, and environmental degradation, it is imperative
that we embrace innovative technologies and strategies to manage our waste more efficiently
and sustainably.
Trash pickup systems, ranging from traditional municipal services to advanced automated
solutions, help keep our communities clean and hygienic, ensuring a healthier and more
pleasant living environment. Moreover, by reducing litter and waste in public spaces, they
contribute to the preservation of our natural habitats, protecting wildlife and ecosystems from
the harmful effects of pollution.
The integration of detection technologies, such as sensors and machine learning algorithms,
into waste management processes offers a promising avenue for improving efficiency and
reducing environmental impact. These systems not only enable early detection of overflowing
bins, but they also optimize collection routes, reducing fuel consumption and greenhouse gas
emissions. This not only benefits the environment but also lowers operational costs for waste
management services.
Furthermore, the use of data analytics and real-time monitoring allows for a more data-
driven and responsive approach to waste management. This means that resources can be
allocated more efficiently, and waste collection can be tailored to the specific needs of each
community. Such data-driven systems can even encourage more responsible waste disposal
practices among individuals, fostering a culture of sustainability and environmental awareness.
In summary, the convergence of traditional trash pickup services with cutting-edge detection
technologies represents a significant step forward in our ongoing efforts to create cleaner, more
sustainable, and environmentally conscious communities. The collective action of governments,
businesses, and individuals, in tandem with the advancement of these technologies, will
undoubtedly contribute to a brighter and more sustainable future. By harnessing the power of
innovation and data-driven decision-making, we can take meaningful strides towards
addressing the challenges of waste management in the 21st century and leaving a cleaner,
healthier planet for generations to come.
Acknowledgements
The authors would like to thank the Armed Forces of Ukraine for providing security to perform
this work. This work has become possible only because of the resilience and courage of the
Ukrainian Army.
References
[1] Dr. Rafique Anjum, Bilal Ahmed Wani, The Concept of Ecological Balance and Environmental
conservation: An Islamic perspective. Suraj Punj Journal For Multidisciplinary Research, (2020),
vol. 8, no. 12, pp. 45-58.
[2] V. Kovtun, I. Izonin, and M. Gregus, Model of functioning of the centralized wireless
information ecosystem focused on multimedia streaming, Egyptian Informatics Journal, vol. 23,
no. 4. Elsevier BV, pp. 89–96, Dec. (2022). doi: 10.1016/j.eij.2022.06.009.
[3] D. Chumachenko, P. Piletskiy, M. Sukhorukova, T. Chumachenko, Predictive Model of Lyme
Disease Epidemic Process Using Machine Learning Approach, Applied Sciences
(Switzerland), 12 (9), art. no. 4282,(2022), doi: 10.3390/app12094282.
[4] V. Semkovych, V. Shymanskyi, Combining OCR Methods to Improve Handwritten Text
Recognition with Low System Technical Requirements. In: Hu, Z., Wang, Y., He, M. (eds)
Advances in Intelligent Systems, Computer Science and Digital Economics IV. CSDEIS 2022.
Lecture Notes on Data Engineering and Communications Technologies, (2023), vol 158.
Springer, Cham. doi:10.1007/978-3-031-24475-9_56.
[5] Jamil Abedalrahim Jamil Alsayaydeh, Azwan Aziz, A. I. A. Rahman, Syed Najib Syed Salim,
Maslan Zainon, Zikri Abadi Baharudin, Muhammad Inam Abbasi and Adam Wong Yoon
Khang, Development of programmable home security using GSM system for early prevention,
ARPN Journal of Engineering and Applied Sciences, (2021), 16(1):88-97.
[6] W.A. Indra et. al., Development of Security System Using Motion Sensor Powered by RF
Energy Harvesting. 2020 IEEE Student Conference on Research and Development, (2020),
SCOReD 2020 9250984, pp. 254-258.
[7] C. Zhihong, Z. Hebin, W. Yanbo, L. Binyan and L. Yu, A vision-based robotic grasping system
using deep learning for garbage sorting, 2017 36th Chinese Control Conference (CCC), Dalian,
China, (2017), pp. 11223-11226, doi: 10.23919/ChiCC.2017.8029147.
[8] S. S. Chandra, M. Kulshreshtha, P. Randhawa, A Review of Trash Collecting and Cleaning
Robots, 2021 9th International Conference on Reliability, Infocom Technologies and
Optimization (Trends and Future Directions) (ICRITO), Noida, India, (2021), pp. 1-5,
doi: 10.1109/ICRITO51393.2021.9596551.
[9] B. Ramalingam et al., Optimal selective floor cleaning using deep learning algorithms and
reconfigurable robot hTetro, (2022), Sci Rep 12, 15938. doi.org/10.1038/s41598-022-19249-7.
[10] L.-B. Chen, X. -R. Huang, et. al., Design and Implementation of an Artificial Intelligence of
Things-Based Autonomous Mobile Robot System for Cleaning Garbage, in IEEE Sensors
Journal, (2023), vol. 23, no. 8, pp. 8909-8922. doi: 10.1109/JSEN.2023.3254902.
[11] Ejour Kulshreshtha, Medhasvi Chandra, et. al. Outdoor Autonomous Trash-Collecting Robot
Design Using YOLOv4-Tiny Electronics, (2021), 10(18):2079-9292.
[12] S. Hossain, B. Debnath, et. al. Autonomous Trash Collector Based on Object Detection Using
Deep Neural Network, TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON), Kochi,
India, (2019), pp. 1406-1410, doi: 10.1109/TENCON.2019.8929270.
[13] J. Bai, S. Lian, Z. Liu, K. Wang and D. Liu, Deep Learning Based Robot for Automatically
Picking Up Garbage on the Grass, in IEEE Transactions on Consumer Electronics, (2018), vol.
64, no. 3, pp. 382-389, doi: 10.1109/TCE.2018.2859629.
[14] B. Ramalingam, A.K. Lakshmanan, et. al. Cascaded Machine-Learning Technique for Debris
Classification in Floor-Cleaning Robot Application. Appl. Sci., (2018), 8, 2649.
doi:10.3390/app8122649.
[15] B. Ramalingam, A.A. Hayat, et.al. Deep Learning Based Pavement Inspection Using Self-
Reconfigurable Robot. Sensors, (2021), 21, 2595. doi:10.3390/s21082595.
[16] I. Izonin, R. Tkachenko, I. Krak, O. Berezsky, I. Shevchuk, S.K., A cascade ensemble-learning
model for the deployment at the edge: case on missing IoT data recovery in environmental
monitoring systems. Front. Environ. Sci., (2023), 11:1295526. doi: 10.3389/fenvs.2023.1295526.
[17] L. Mochurad, Y. Hladun, R. Tkachenko, An Obstacle-Finding Approach for Autonomous
Mobile Robots Using 2D LiDAR Data. Big Data and Cognitive Computing. (2023), 7(1):43.
doi:10.3390/bdcc7010043.
[18] J. Zhang, Z. Chen, et. al., Faster and Lightweight: An Improved YOLOv5 Object Detector for
Remote Sensing Images. Remote Sensing, (2023), 15(20):4974.
doi.org/10.3390/rs15204974.
[19] TACO dataset. Available online: http://tacodataset.org/explore (accessed on 3 November 2023).