In the following paper, we propose an alternative approach to automatic recognition and sorting of the agricultural objects. The objects are sorted, based on their color. The sorting algorithm works with pictures, captured by a web camera, pre-processes them, determines the mass centers of the recognized objects, and calculates distances to the points which were found. Afterward, the recognized objects are sorted by a hand-manipulator with four degrees of freedom. The manipulator is powered by Arduino Uno Rev 3 and servo motors of two types. The sorting algorithm has been implemented in Python, with the use of the OpenCV library. For image pre-processing, the spatial anti-aliasing, filtering, and morphological opening and closing have been used. Color maps have been created to determine the mass centers of the recognized objects. To calculate the rotation angle of the manipulator's elbow based on the shoulder position, linear regression model is trained. The practical value of the research and development results is the possibility of their use in the harvesting of the agricultural objects with further extend to manufacture with conveyor sorting tapes.
As one of the key branches of the industry of Ukraine, agricultural industry aims to deliver the high-quality goods with lower self-costs, while also reducing the time spent on their production. Replacing the manual labor with the machine labor is an effective way of solving this problem. A human laborer cannot perform repetitive tasks for a long time and stay effective for long.
Since harvesting is one of the primary parts of all agricultural processes, automating this task has been a researchers’ priority since the 1960s. However, automated harvesting has several complications to avoid. Accurate recognition of objects and avoiding their damage during the gathering process are the main problems for every researcher aiming to automate this procedure. ___________________ Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
One of the first researches devoted to automatic harvest recognition and gathering
was performed in 1968 [
Since the problem of the automatic agricultural objects’ sorting comprises both
object recognition and capturing, it encounters the same complications as the
automatic harvesting task does. Nowadays, there are several approaches to the
automatic sorting of the agricultural objects – magnetohydrodynamic installations
[
In this work, we propose a comparatively low-complexity and effective approach to the automatic agricultural objects’ sorting. 2
In this section, we will consider some of the existing methods of the automatic sorting of the agricultural objects, based on the machine vision. 2.1
Most of the modern researches use images, captured by CCD cameras, for object
detection and recognition. Some of them rely on the fuzzy logic algorithms [
Bayesian classifiers [
Clustering methods [
Neural networks [
All the methods listed above are effective but quite significant amount of time and resource consuming. In this work, we propose an alternative approach to recognition and sorting of the agricultural objects. 2.2
There are several criteria for choosing the hand-manipulator prototype: ─ a convenient gripper, that would not damage the captured objects; ─ minimum 4 degrees of freedom; ─ Arduino-compatibility (for reducing the cost of the project).
Wireless Robot Arm [
Niryo One from Niryo is an analog of an industrial robot, constructed for learning purposes. It has six degrees of freedom and is made of 3D-printed details. Niryo One is an open-source model, and has its own IDE with a drag-and-drop editor, which makes a model available for all users. The manipulator can be powered by both Arduino and Raspberry PI.
The model also has several grippers which can be mounted into it. But the huge variety of details makes Niryo One complicated to construct and maintain.
Dexter by Haddington Dynamics [
The uArm by UFACTORY is powered by Arduino MEGA 2650 and has four degrees of freedom. Each of the joints can rotate on 180°. The gripper has a round form and is accustomed for capturing round objects. The model is an open-source project, so the manufacturer provides the drawings of the details, which can be carved from acryl to reduce the budget of the project.
Having compared the above-mentioned manipulator models, we decided to perform the reverse engineering of uArm by UFACTORY. This hand-manipulator can lift objects up to 0.5 kg in weight, has a 320 mm working area, and is Arduinocompatible by default. 3
In this work, the hand-manipulator, powered by Arduino Uno Rev3 and two types of TowerPro servomotors, is used. The object recognition algorithm implemented in Python with the use of OpenCV library, determines the rotation angle for the manipulator’s shoulder and elbow. The manipulator rotates on the appropriate angle, captures the object, and puts it either on the left or on the right, based on its color. 3.1
The constructed manipulator has the following characteristics: ─ height – 300 mm; ─ working area – 140-300 mm; ─ carrying capacity – 200 g; ─ input current – 6 A; ─ board – Arduino Uno Rev 3; ─ connection method – MicroUSB.
All the movable joints of the manipulator have bearings on them, which makes the moves faster and more coordinated. More powerful servomotors (TowerPro MG995) are located in the base, shoulder and elbow of the manipulator, which makes the construction more stable.
The function of Arduino Uno Rev 3 lies in getting the value of the rotation angle for the manipulator’s shoulder through the COM port, and sending the respective commands to the servomotors. The Arduino is connected to a computer where the sorting algorithm runs, and is powered by it. For servomotors, the external source of power is used (5 V, 5 A). Fig. 2 shows the connection scheme of the overall complex.
Five servomotors of two types run the hand-manipulator. Three TowerPro MG995 servomotors are placed in the base, elbow, and shoulder of the manipulator, with the torque of 10 kg*cm with the 6 V power supply. The rotation speed with such a power supply is 0,16s/60º. The gripper opening-closing and rotating are performed by the TowerPro MG90S servomotors. Their torque is 2.2 kg*cm with the 6V power supply, and the rotation speed is 0,08s/60º. Both models of servomotors work with any type of the Arduino output (the PWM mode is not obligatory).
The algorithm that controls Arduino can be described by a finite Mealy machine, shown on Fig. 3. In case the server sends an inappropriate signal to Arduino, it just keeps waiting for the next command to be read. 3.2
The object detection algorithm was described on Python with the use of the OpenCV
library [
The algorithm is meant to sort agricultural objects of two colors – yellow and red. For more convenient recognition of the manipulating hand itself, the green mark was put on the gripper.
First, the algorithm looks for all red objects and moves them to the left. Then, it starts looking for yellow objects and moves them to the right. When all the objects are sorted, the algorithm stops working.
Fig. 4 shows the statechart diagram for the algorithm. To increase the algorithm performance, the image preprocessing starts even before a user starts working with a program. The image preprocessing aims to reduce noise on the image and consists of spatial anti-aliasing, filtering (color maps creation), and morphological opening/closing.
For the spatial anti-aliasing, the mean filtering method is used. The 9x9 filter was used, with the value of each sell being equal to 1/81. Since the input image had the BGR representation, the filter was put on the blue, green and red channels separately. Fig. 5 shows an image before and after the mean filtering. After applying the mean filter, the image was converted to the HSV-format. The model is a non-linear representation of the RGB-model.
The next stage of an algorithm is creating the color maps for yellow, red, and green colors by highlighting all the pixels of a particular color. The pixel’s belonging to the following of HSV-values determined its inclusion on a particular color map: ─ green – from (45, 50, 50) to (75, 255, 255); ─ yellow – from (20, 100, 100) to (30, 255, 255); ─ red – from (0, 70, 50) to (10, 255, 255) and from (170, 70, 50) to (180, 255, 255). To every color map, morphological opening and closing were applied. Fig. 6 shows the outcome of these operations.
a) b) c) On the resulting color maps, the mass centers of the objects are determined. For each map, both horizontal and vertical projections of the colored areas are made. Afterward, the means along each axis are calculated. The expected value of the center coordinate is calculated by the formula
E x f (x) f (x) q dx , (1) where q – 90% quantile.
To determine the mass centers of the objects more precisely, the exponential moving average method is used. The method is determined by the formula c ct1 (1 ) xc , where – anti-aliased mass center coordinate, ct–1 – previous value of an anti-aliased mass center coordinate, xc – not anti-aliased mass center coordinate, α – anti-aliasing coefficient.
For the coordinates of the gripper, the moving average formula is used v 1 n1
vt 1 , n 0 (2) (3) where v — anti-aliased mass center coordinate, n – anti-aliasing interval, vt–1 – the previous value of the anti-aliased mass center coordinate.
In the program interface, the mass centers before anti-aliasing are shown in red, while the mass centers after anti-aliasing are showed in green (see Fig. 7, a). a) b) To determine the least distances to the objects, the manipulator scans all the area, captured by a web camera (the base rotates from 0º to 130º). On each iteration, the rotation angle of the base changes on 2º. The mass centers are re-calculated on each iteration and the Euclidian distance between the mass centers of the gripper and the object are calculated.
When the scanning process is over, the least Euclidian distance from each object is taken from the array, along with the appropriate base rotation angle. To capture the object, the manipulator rotates on the respective angle. Fig. 7, b shows the moving trajectories of the objects during the scanning and capturing process.
During the capturing process, it is important to keep the balance between the manipulator’s shoulder and elbow to keep the construction stable. For this purpose, the linear regression model was trained. To capture the object successfully, the gripper should stay approximately 2 cm high from the surface. The dataset for the linear regression model consisted of 20 combinations of the shoulder’s and elbow’s rotation angles, which guaranteed that the gripper stayed on the needed distance from the surface. The formula for determining the elbow position based on the shoulder rotation angle is pl 13 pr 1, 54 (4) When the manipulator aims to capture the object, the Euclidian distance between the objects and the gripper is calculated on each iteration. If the distance is bigger than 100, the shoulder position changes on 1º. The elbow position is calculated with the formula (4). When the distance becomes less than 100, the object is captured. 4
The complex was tested during the test session of 30 experiments, with the objects being placed on distances between 20 cm and 30 cm from the gripper. In each experiment, the yellow and red test objects were interchanged.
Testing was conducted in the same circumstances: on the white background, under the same lighting. The starting position of the hand-manipulator was fixed, along with the position of the camera. The accuracy of the object recognition is 93,4%. The accuracy of the object capturing and moving is 78,3%. Fig. 8 shows the testing results for the red objects.
Testing showed that the distance between 26 cm and 30 cm between the manipulator and objects is optimal for successful sorting. The model of investigated multi-agent system (MAS) for efficient sorting based on the autonomous joints rotation is (5) kb kc fp a cos ; n n i j1 1 i jr1 1 xi cos( fp) m1 n m j n i j1 1 i jr1 1 yi m1 n sin( fp) m j n SMAS M (xi , yi , z const).
i 1 cos( fp) m jr n
i 1 sin( fp) m jr n cos( fp); sin( fp); where i – the infinitesimal segment along the manipulator, a – the angle of the tip of the manipulator’s base with respect to the horizontal axes, b – is the number of periods in a length, and c – the direction of movements for SMAS as handler function.
During the experimental tests an idea about multiple hand-manipulators combined as multi-agent system was released. According to the need for several handmanipulators to be applied to the sorting process, the problem of intersections between them is become important. Every agent ASMAS should include the minimal behavior’s algorithms based on the crossing collisions index r of multiple segments of the hand-manipulators. 5
In the current paper, we present the hardware and software parts of the device for automatic harvest sorting. The hardware part of the complex is the hand-manipulator with four degrees of freedom, controlled by the Arduino Uno Rev 3. The software part of the complex is the algorithm, described on Python with the use of the OpenCV library. The algorithm works with images at 10 FPS and searches for mass centers of the determined objects less than 0,08 s. The accuracy of the object recognition is 93,4%. The accuracy of the object capturing and moving is 78,3%.
The significant advantage of the system is that the hardware part doesn’t damage objects during the sorting process. The disadvantages of the tested device are the low speed of the hand-manipulator, and comparatively low accuracy of the objects’ capturing and moving. The system can be used on the enterprises with the conveyor tapes, used for harvest sorting.