In this paper, we propose a tele-operated robotic system combined Robot Operating System (ROS) 2 and Internet of Things (IoT), aiming to solve labor shortage problem in Japanese food industry. The proposed robotic system consists of a ROS 2 based motion control module, a You Only Look Once (Yolo) V5 based object recognition module, and an Amazon Web Service (AWS) based cloud computing module. To show the efectiveness of the proposed robotic system, several typical pick-and-place tasks were conducted for handling three fruit samples such as apple, orange, and strawberry. These food samples could be successfully grasped and placed to desired destinations using the proposed robotic system.
In Japanese food industry, the labor shortage caused by the aging society has become a social
problem [
The robotic motion control module utilizes the MoveIt 2 package of ROS 2 for robotic motion control. which includes a series of useful functions, such as motion planning, manipulation, three-dimensional (3D) perception, kinematics, control, and navigation[13]. The desired control commands was transmitted via USB to Dynamixel protocol between the local PC and this module.
The object recognition module utilizes the Yolo V5 for object detection. Yolo is a state-ofthe-art, real-time object detection algorithm. The bounding boxes of the target object can be predicted via a single neural network, and be classified with certain probabilities. Using Yolo, real time object detection could be realized due to its fast process rate. In addition, YOLO also has high detection accuracy with few background errors [14]. Although other versions of Yolo packages can be used, this time we selected Yolo V5 in our preliminary experiments.
The cloud computing module utilizes the AWS for communications between remote PC and local PC via Message Queuing Telemetry Transport (MQTT) protocol. AWS ofers a broad set of Internet of Things (IoT) services that allow devices to connect to the cloud and interact with other devices and cloud applications. The AWS IoT Core is a managed cloud service that enables connected devices securely interact with cloud applications and other devices. It can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely [15].
To show the efectiveness of the proposed system, several pick-and-place tasks were conducted for handling a series of food samples, such as apple, orange, and strawberry.
Local PC
Camera Plate
Handled Object
Robot
Remote PC
be successfully picked up and be placed to the desired plate. The corresponding recognition confidences were 0.90 and 0.70, and the successful rates of these tasks were 100%, as shown in Table 1.
In this study, we developed a remote controlled robotic system combined ROS2 and IoT. The proposed system was successfully applied for typical pick-and-place tasks. A series of food samples could be successfully picked up and be placed to desired destinations using the proposed system. In the future, we would like to conduct the pick-and-place tasks using actual food products. Furthermore, monitoring is commonly required for the remote control system. However, the current system could not remotely monitor the sensor data from the robot side, which will also be solved in our future study. In addition, it is believed that hardware resources of the proposed system could be reduced by utilizing the resources of cloud services and performing calculations such as recognition and trajectory calculations on the cloud side. [9] D. Kang, Y. Oh, J. H. Bong, S. Park, S. Kim, Development of augmented reality-based display for tele-operation robotic system: Remote control of robotic system with enhanced display for the operator, 2016. URL: https://api.semanticscholar.org/CorpusID:64562928. [10] S. Nisar, O. Hasan, State of the art and key design challenges of telesurgical robotics, Advanced Methodologies and Technologies in Artificial Intelligence, Computer Simulation, and Human-Computer Interaction (2019). URL: https://api.semanticscholar.org/CorpusID: 115365492. [11] P. Picozzi, U. Nocco, G. Puleo, C. Labate, V. Cimolin, Telemedicine and robotic surgery: A narrative review to analyze advantages, limitations and future developments, Electronics (2023). URL: https://api.semanticscholar.org/CorpusID:266603412. [12] E. Bertino, D. Bliss, D. Lopresti, L. Peterson, H. Schulzrinne, Computing research challenges in next generation wireless networking, 2021. arXiv:2101.01279. [13] D. Coleman, I. Sucan, S. Chitta, N. Correll, Reducing the barrier to entry of complex robotic software: a moveit! case study, arXiv preprint arXiv:1404.3785 (2014). [14] G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, NanoCode012, Y. Kwon, K. Michael, TaoXie, J. Fang, imyhxy, Lorna, Z. Yifu, C. Wong, A. V, D. Montes, Z. Wang, C. Fati, J. Nadar, Laughing, UnglvKitDe, V. Sonck, tkianai, yxNONG, P. Skalski, A. Hogan, D. Nair, M. Strobel, M. Jain, ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation, 2022.
URL: https://doi.org/10.5281/zenodo.7347926. doi:10.5281/zenodo.7347926. [15] Aws, 2024. Https://aws.amazon.com/jp/.