=Paper=
{{Paper
|id=Vol-2761/HAICTA_2020_paper18
|storemode=property
|title=Identifying the Technological Needs for Developing a Grapes Harvesting Robot: Operations and Systems
|pdfUrl=https://ceur-ws.org/Vol-2761/HAICTA_2020_paper18.pdf
|volume=Vol-2761
|authors=Eleni Vrochidou,Theodore Pachidis,Michail Manios,George Papakostas,Vassilis Kaburlasos,Serafeim Theocharis,Stefanos Koundouras,Katerina Karabatea,Elizabeth Bouloumpasi,Stavros Pavlidis,Spyridon Mamalis,Theodora Merou
|dblpUrl=https://dblp.org/rec/conf/haicta/VrochidouPMPKTK20
}}
==Identifying the Technological Needs for Developing a Grapes Harvesting Robot: Operations and Systems==
https://ceur-ws.org/Vol-2761/HAICTA_2020_paper18.pdf
Identifying the Technological Needs for Developing a
Grapes Harvesting Robot: Operations and Systems
Eleni Vrochidou1, Theodore Pachidis2, Michail Manios3, George A. Papakostas4,
Vassilis G. Kaburlasos5, Serafeim Theocharis6, Stefanos Koundouras7, Katerina
Karabatea8, Elizabeth Bouloumpasi9, Stavros Pavlidis10, Spyridon Mamalis11,
Theodora Merou12
1Human-Machines Interaction Laboratory (HUMAIN-Lab), International Hellenic
University, Agios Loukas, 65404 Kavala, Greece; e-mail: evrochid@teiemt.gr
2Human-Machines Interaction Laboratory (HUMAIN-Lab), International Hellenic
University, Agios Loukas, 65404 Kavala, Greece; e-mail: pated@teiemt.gr
3Human-Machines Interaction Laboratory (HUMAIN-Lab), International Hellenic
University, Agios Loukas, 65404 Kavala, Greece; e-mail: m.manios@teiemt.gr
4Human-Machines Interaction Laboratory (HUMAIN-Lab), International Hellenic
University, Agios Loukas, 65404 Kavala, Greece; e-mail: gpapak@teiemt.gr
5Human-Machines Interaction Laboratory (HUMAIN-Lab), International Hellenic
University, Agios Loukas, 65404 Kavala, Greece; e-mail: vgkabs@teiemt.gr
6School of Agricultural Biotechnology and Oenology, International Hellenic University,
Drama, 66100 Greece; e-mail: sertheo@agro.auth.gr
7School of Agricultural Biotechnology and Oenology, International Hellenic University,
Drama, 66100 Greece; e-mail: skoundou@agro.auth.gr
8School of Agricultural Biotechnology and Oenology, International Hellenic University,
Drama, 66100 Greece; e-mail: katerina_karampatea@yahoo.gr
9School of Agricultural Biotechnology and Oenology, International Hellenic University,
Drama, 66100 Greece; e-mail: bouloumpasi@gmail.com
10School of Agricultural Biotechnology and Oenology, International Hellenic University,
Drama, 66100 Greece; e-mail: stavrospavlides@yahoo.gr
11School of Agricultural Biotechnology and Oenology, International Hellenic University,
Drama, 66100 Greece; e-mail: mamalis@teiemt.gr
12School of Agricultural Biotechnology and Oenology, International Hellenic University,
Drama, 66100 Greece; e-mail: thmerou@teiemt.gr
Abstract. Robots are increasingly entering agricultural fields to support human
labor. Heavy tasks like harvesting are assigned to robots due to their advanced
modularity, robustness and accuracy that provide automated solutions to tedious
and elaborate tasks. In this paper, the technological requirements of a specialized
Agrobot (Agriculture robot) for supporting viticulture tasks such as harvest,
green harvest and defoliation, are identified. This robot aims at developing on-
board intelligent decision making on-the-spot based on commercial hardware,
machine vision and innovative computational intelligence algorithms. Design,
structures, methods and sensors are briefly discussed. This study delineates a
prototype grape harvesting robot, consisting of a reliable information acquisition
system that includes sensor-fusion algorithms and data analysis, adopted to the
dynamic conditions of agricultural environments such as vineyards.
105
Keywords: Harvesting robot; Agrobot; automation; intelligent system; system
design.
1 Introduction
Current practices of grape harvest include human involvement, such as monitoring
and specialized manual work, which seems to have reached its limits; young people
abandon viticulture because of inherent work difficulties, the increasing age of grape
harvesters prolongs harvest duration, therefore reduces the quality of the harvested
grapes and of the produced wines. Thus, there is a need to reliably automate grape
harvest (Avital Bechar, 2010; Mavridou et al., 2019). Vine harvester combines were a
first attempt toward automation. However, the advent of Agrobots has the potential to
raise the quantity and quality of grape product in a constant way, reduce production
costs and manual labor and compensate for the shortage of specialized workers.
Extensive research has focused on the application of Agrobots to a variety of
vineyard tasks (Bac et al., 2014). Kondo (Kondo, 1991) developed a trial robot to
harvest individual bunches of grapes, consisted of a hand which could hold and cut
rachis, a visual sensor and a crawler type travelling device. Monta et al. (Monta, Kondo
and Shibano, 1995) constructed a multipurpose robot for vineyard tasks, consisted of
a manipulator, a visual sensor, a traveling device and end-effectors, able to perform
several tasks. A research team (Morris, 2007) developed a total vineyard
mechanization system of almost all practices, including dormant and summer pruning,
leaf removal, shoot and fruit thinning, canopy management, and harvesting. Two
harvesting machines functioning under the same principles were developed by Pezzi
et al. (Pezzi, Balducci and Pari, 2013); one with horizontal shaking and another with
vertical shaking. The machines were tested and compared.
Simplifying the agricultural tasks and enhancing Agrobots with sensors may
improve their performance, but this will refer to a non-feasibly constructed Agrobot
and it is insufficient for successful implementation in real in-field practices. For this
reason, robotics and non-robotics disciplines need to be identified (Burks et al., 2005)
so as to address the following requirements: the robot has to be technically capable of
performing the defined agricultural tasks, economically feasible, safe and accepted by
the farmers. To comprise all requirements successfully, a multidisciplinary team of
specialized experts including engineers, viticulturists and agronomists should extract
design methods collaboratively. In this work, the technological needs for the
development of an “intelligent” wheeled Autonomous Robot for Grape harvest (ARG)
are identified. The objective of this study is to delineate optimally the ARG
requirements by a group of experts, towards mechanizing grape harvest in a way that,
instead of massively harvesting rows of vines, to harvest selected grapes from the
vineyard. This study is based on the ongoing work of project POGHAR (Personalized
Optimal Grape Harvest by Autonomous Robot)(Personalized Optimal Grape Harvest
by Autonomous Robot (POGHAR), 2018). POGHAR regards the development of an
autonomous ground robot equipped with a robotic arm to support viticulture tasks such
as harvest, green harvest and defoliation.
106
The rest of the paper is structured as follows. In Section 2 the requirements of the
selected agricultural operations are defined. According to these, in Section 3 the
corresponding technological needs of the robot, in terms of hardware and software are
identified. Conclusions are presented in Section 4.
2 Grapes Harvest Activity Requirements
First, defoliation will be automated, followed by green harvest in order to optimize
grape quality/quantity and prepare vines for the harvest. Finally, homogeneous harvest
is automated in the sense that ARG will harvest only grapes of similar degree of
ripeness. In what follows, the requirements for the three agricultural tasks are
addressed by a group of viticulturists and agronomists.
2.1 Defoliation
Selectively removal of leaves, namely defoliation, is suggested in order to improve
the microclimate in vineyards (Fig.1). Thus, the quality of the produced wine is
increased due to better plant health i.e. better ventilation and due to acquisition of better
phenolic characteristics of grapes as a result to their exposure to the sun. The robot
must remove carefully and uniformly a percentage of leaves on the crop base, only
from the east side of the vineyard.
(a) (b) (c)
Fig. 1. A vineyard (a) before, (b) during and (c) after defoliation.
2.2 Green Harvest
In order to improve the phenolic content of grapes, it is necessary to reduce the load
on the vine; a number of grapes are removed from each tree. This process, namely
green harvest, meliorates taste and aroma of the remaining grapes (Fig.2). The robot
must remove a percentage of grape clusters from each vine, in a priority order as
follows: sick, malnourished, uneven and immature.
107
2.3 Harvest
When grapes are ripened, harvesting takes place. At this phase, the robot must
remove all ripened grape clusters (Fig.3) and place them in harvesting baskets.
Sick/damaged clusters are not collected and either removed or left on the vine.
3 Defining the Technological needs of ARG
The in-field robot must be able to understand the physical properties of each
encountered object and be able to work under dynamic conditions. The robot first will
acquire raw data about the environment from its sensors, will analyze it for reasoning
and will operate based on its perception according to its operation plan.
(a) (b)
Fig. 2. Green harvest for (a) a red and (b) a white grape variety.
(a) (b)
Fig. 3. Homogenous harvest. Red grapes (a) not fully ripened and (b) fully ripened ready to
harvest.
Technically, the ARG will be developed by the integration of a wheeled robot, one
robot arm, end-effectors such as a cutter, electronic sensors e.g. cameras, and software
108
that will coordinate the operation of the mechanical and electronic devices. In addition,
an aerial drone could contribute to the development of digital maps required for ARG’s
autonomous navigation. ARG’s sensing system needs to be equipped with specialized
manipulators and end-effectors able to work under varying conditions, since dust,
humidity, heat etc. can easily affect the electric circuits and cause corrosion and,
therefore, instability to the system. Regarding reasoning and planning, effective
software needs to be developed so as to ensure two basic abilities for ARG; robot
functionalities (e.g. obstacle avoidance, self-localization, path planning, navigation)
and robot applications (e.g. harvesting, defoliating, handling). Algorithms need to be
adaptive and able to deal with object and environment variations such as illumination.
In what follows, a group of electrical engineers and computer scientists addresses the
hardware (Fig.4(a)) and software (Fig.4(b)) needs for developing the ARG.
3.1 Hardware Needs
Drone. An unmanned aerial vehicle (UAV) will acquire multiple RGB imagery of
vineyards, resulting in a large fraction of overlapping between them to derive a binary
difference dense model (DDM). The DDM will provide the 3-D macro structure of the
vineyard which is necessary for the navigation of ARG.
(a) (b)
Fig. 4. Technological needs in (a) Hardware and (b) Software.
Wheeled Robot. A mobile robotic platform, especially designed for outdoor missions
and able to navigate in different types of terrains, will carry all sensors around the field
and necessary electronics. It needs to be equipped with sensors such as Global
Positioning System (GPS), Light Detection and Ranging (LIDAR), Inertial
Measurement Unit (IMU), 3D camera and encoders to facilitate localization,
navigation and obstacle avoidance. It needs to be equipped also with an extra battery
109
ensuring the proper operation of the system for a long time, voltage converters and
signal processing units.
Manipulator. A robotic arm will perform all agricultural operations. The arm should
be lightweight, at least of 6 Degrees-Of-Freedom (DOF) and able to carry the weight
of a big grape cluster. The arm will be equipped with adequate end-effectors such as
two fingers to hold a stem and a cutter to cut it off.
Sensors. Cameras are the most important external sensors for a harvesting robot as they
replace human’s eyes for discrimination, recognition and distance measurements. The
two RGB cameras mounted on the robotic arm will provide stereoscopic vision and a
near infrared (NIR) and a thermal camera will provide additional information for the
spectral characteristics of vines.
3.2 Software Needs
Traceability and Geo-positioning. The Agrobot must navigate between the lines of the
vineyard and sample. The maps of the vineyards must first be extracted from the drone
information (Fig.5(a)). Then, a wireless positioning system based on GPS needs to be
developed so as to provide positioning information of the robot and use it to support
real-time precision navigation between specific geographic coordinates of the map.
Navigation and Guidance. An algorithm must be developed to extract the navigation
path in the vineyard (Fig.5(b)), including the headland turns between the vineyard
lines. The algorithm needs to combine machine vision and both global and local
positioning information from the mounted sensors, so as to determine the navigation
paths, and execute path planning along with obstacle avoidance.
(a) (b)
Fig. 5. (a) Extracted grapevine map from multiple UAV RGB images and (b) robot’s trajectory
between the lines (red color) of the vineyard.
Computer vision. Computer vision algorithms (Fig.6), based on in-field images
(Fig.7(a)), need to be developed to support the following tasks:
110
1) Leaves detection (Fig.7(b)) so as to define the position and percentage of leaves to
be removed during the defoliation process.
2) Grape clusters detection (Fig.7(c)) so as to define the position and percentage of
grape clusters to be removed during green harvest, giving priority to the
malnourished, uneven and immature.
3) Ripeness estimation during harvest. Only fully ripened clusters will be removed.
The rest will remain on the vines or removed separately from the healthy ones.
4) Stem detection (Fig. 7(d)). The stem of leaves and grape clusters needs to be
detected, approached and cut-off by the end-effector.
5) Harvesting baskets detection. The baskets need to be located and filled with the
collected clusters up to a certain point.
6) Movement of the robotic arm toward a target point. Stereoscopic vision will be
used to locate the exact distance of an object.
7) Movement of the robotic arm by avoiding collisions. Cameras mounted on the
robotic arm will be used as real-time collision detectors. Manipulators and sensors
must be able to cope with various geometries of obstacles and targets. Detection
and motion planning algorithms have to generate new manipulator trajectories
adapted to every target in a short time horizon.
Fig. 6. Computer vision algorithms need to be implemented into the ARG.
111
(a) (b)
(c) (d)
Fig. 7. (a) Original RGB image and segmented grape (b) leaf, (c) cluster, (d) stem.
User Interface. A user-friendly interface needs to be developed for establishing the
communication between user and robot. The user needs to: have access to the vineyard
map, create the navigation path, select exact points on the map where the Agrobot will
perform selected tasks and define the tasks. The user interface should permit
personalized changes for each agricultural task, e.g. percentage of leaves/clusters to
remove. Moreover, it must provide monitoring information for the robot, e.g. battery
level, current position, status of sensors and live streaming.
4 Conclusions
This work addresses the technological needs for the development of an Agrobot that
deals with three vineyard tasks; defoliation, green gravest and harvest. The objective
of this study is to delineate optimally the ARG requirements by a group of experts,
towards mechanizing effectively the aforementioned tasks. Identifying the
technological needs of ARG contributes to a more realistic system design. The
literature indicates that systematic design process techniques can contribute to the
112
technical and economic feasibility of a robot (Angeles and Park, 2008). After
identifying the needs, the next phase of the ARG development is the electric and
mechanical connection of sensors combined with effective algorithms (Kaburlasos et
al., 2019) towards implementing the above-mentioned agricultural tasks.
Acknowledgments. This research has been co-financed by the European Union and
Greek national funds through the Operational Program Competitiveness,
Entrepreneurship and Innovation, under the call RESEARCH – CREATE –
INNOVATE (T1EDK-00300).
References
1. Angeles, J. and Park, F. C. (2008) ‘Performance Evaluation and Design Criteria’,
in Springer Handbook of Robotics. Berlin, Heidelberg: Springer Berlin Heidelberg,
pp. 229–244. doi: 10.1007/978-3-540-30301-5_11.
2. Avital Bechar (2010) ‘Robotics in horticultural field production’, Stewart
Postharvest Review, 6(3), pp. 1–11. doi: 10.2212/spr.2010.3.11.
3. Bac, C. W. et al. (2014) ‘Harvesting Robots for High-value Crops: State-of-the-art
Review and Challenges Ahead’, Journal of Field Robotics, 31(6), pp. 888–911.
doi: 10.1002/rob.21525.
4. Burks, T. et al. (2005) ‘Engineering and Horticultural Aspects of Robotic Fruit
Harvesting: Opportunities and Constraints’, HortTechnology, 15(1), pp. 79–87.
doi: 10.21273/HORTTECH.15.1.0079.
5. Kaburlasos, V. G. et al. (2019) ‘Time Series Classification in Cyber-Physical
System Applications by Intervals’ Numbers Techniques’, in 2019 IEEE
International Conference on Fuzzy Systems (FUZZ-IEEE). New Orleans: IEEE,
pp. 1–6. doi: 10.1109/FUZZ-IEEE.2019.8858946.
6. Kondo, N. (1991) ‘Study on grape harvesting robot’, IFAC Proceedings Volumes,
24(11), pp. 243–246. doi: 10.1016/B978-0-08-041273-3.50047-1.
7. Mavridou, E. et al. (2019) ‘Machine Vision Systems in Precision Agriculture for
Crop Farming’, Journal of Imaging, 5(12), p. 89. doi: 10.3390/jimaging5120089.
8. Monta, M., Kondo, N. and Shibano, Y. (1995) ‘Agricultural robot in grape
production system’, in Proceedings of 1995 IEEE International Conference on
Robotics and Automation. IEEE, pp. 2504–2509. doi:
10.1109/ROBOT.1995.525635.
9. Morris, J. R. (2007) ‘Development and Commercialization of a Complete Vineyard
Mechanization System’, HortTechnology, 17(4), pp. 411–420. doi:
10.21273/HORTTECH.17.4.411.
10. Personalized Optimal Grape Harvest by Autonomous Robot (POGHAR) (2018)
HUMAIN Lab. Available at: http://evtar.eu/ (Accessed: 16 July 2020).
11. Pezzi, F., Balducci, G. and Pari, L. (2013) ‘The Comparison Between Two Grape
Harvesting Machines With Horizontal and Vertical Shaking Systems’, Acta
Horticulturae, (978), pp. 257–262. doi: 10.17660/ActaHortic.2013.978.30.
113