Multiple leaflets-based identification approach for compound leaf species

Abstract

Leaves of plants can be classi ed as being either simple or compound according to their shapes. Compound leaves can be seen as a collection of simple leaf-like structures called lea ets. However, most computer vision-based approaches describe these two leaf categories similarly. In this paper, we propose a new description and identi cation method for compound leaves that takes into account particularities related to the arrangement of their shapes (speci cally, their division into lea ets). In fact, we propose a new multiple lea ets-based identi cation approach. Our main motivation behind this choice is that some compound leaf species may hold variabilities in terms of their lea ets number, size and even shape. Thus, a local description based on a certain number of lea ets may provide greater accuracy. In our approach, we were limited to three lea ets that were automatically extracted from image based on some geometric assumptions inspired from botany. Then, we construct and evaluate our identi cation scheme based on some classical texture descriptors for local lea ets description and using some state-of-the-art fusion algorithms to combine responses obtained from each lea et query. Experiments carried out on compound leaves of the Pl@ntLeaves scan database have shown an improvement in classi cation results with regard to entire image query.

1. INTRODUCTION

New interdisciplinary technologies that integrate computer vision in botanical research are being developed in response to ecological challenges such as global climate change, rapid urban development, destruction of habitats, overexploitation of natural resources, food insecurity, biodiversity crises, etc. In particular, computer vision studies are increasingly focusing on accurate, complete and user-friendly systems for taxonomic identi cation of plant species (i.e, intended for a wide range of people, not only experts). A number of project-systems have already been built, for instance, Leafsnap in America [ 15, 19 ], CLOVER [ 25 ] in Asia, Pl@ntNet[ 4 ], ReVes [ 5 ] and ENVIROFI [ 1 ] in Europe, etc, and most of these systems use leaves to identify plant species. In fact, unlike other organs such as owers, fruits or seeds, leaves are generally easy to collect (available throughout the year) and to scan or photograph (they have an approximately two-dimensional shape). Moreover, they often hold discriminative information that is useful for characterizing plant species.

Existing leaf-based plant identi cation approaches di er in several aspects: One is the type of feature used. Fundamental features are shape [ 22, 30, 29 ] and texture [ 9, 3, 8 ] which describe respectively the leaf margins and the vein pattern, the main key indicators of leaf species. Another aspect concerns the way the leaf is viewed: using generic or domain-speci c representations. Generic approaches consist in using common computer vision representations such as the Shape Context, the Curvature Scale Space, the MultiScale Fractal representation, the Fourier and Wavelet Transforms [ 8 ]). These methods have the advantage of being simple and rapid. However, they are not always su cient to provide accurate identi cations mainly due to the high inter-class and low intra-class similarity that occur for some species in terms of certain characteristics. For example, in the case of the Acer Negundo (see Figure 1), the use of contour descriptors may induce errors since some specimens have lobed and/or serrated margins while others have entire margins. For that reason, there has been a recent trend toward using domain-speci c or botanical knowledge, particularly about the leaf architecture, in order to enrich the leaf image representation [ 7, 12, 23, 26 ]. In fact, the leaf architecture, built and extensively used by botanists, refers to the description and categorization of leaves according to the properties of their structure. This includes several foliar characters that describe, hierarchically, the form and the placement of di erent elements constituting the leaf structure such as venation pattern (see 1st row of Figure 2) [ 31, 17 ], marginal con gurations (see 2nd row of Figure 2) [ 6 ], shapes of leaf parts (see 3rd row of Figure 2) [ 14 ], etc. So far, the use of this information has remained limited to some simple characters (such as the laminar form described by the ratio of the laminar width and height, the apical and basal form expressed respectively by the apex and base angles, etc. [ 16 ]). In this paper, we are interested in one of the most important characters that has been less exploited: the leaf arrangement (or type). In fact, leaves of trees are grouped into two basic classes: simple and compound leaves (see Figure 3).

In current leaf image retrieval approaches, simple and compound leaves are described similarly by using global descriptors (for the whole leaf image) [ 22, 30, 29, 9, 3 ]. However, compound leaves can be seen as a subdivision of simple leaves. Otherwise, each lea et has a blade-like structure. For that reason, a local description, whose regions of interest are lea ets, may provide greater accuracy, not only in the case of occlusion or partial damage, but more speci cally, when dealing with partial intra-species non-similarity. For instance, the Gleditsia triacanthos species may have di erent types of lea ets (simple and pinnate) within the same leaf (see Figure 4). Also, the fraxinus angustifolia and the fraxinus ornus species may have a variable number of lea ets (see Figure 5). Furthermore, the Vitex Agnus Cactus species hold lea ets with di erent sizes (see Figure 6).

In these cases, the whole leaf images are clearly totally di erent. However, the similarity can be revealed by comparing lea ets separately. From this assertion, in this paper, we propose an image retrieval system for compound leaves based on the combination of response lists derived from each lea et sub-image query. This involves the following steps: First, we automatically detect at most three represen

(b) Fraxinus ornus tative lea ets of the image using geometric properties related to their contours.

Then, we consider sub-images of lea ets as multiple views of the same compound leaf image. In other words, we replace the entire image by its three lea ets in the identi cation scheme. We index each lea et subimage separately. The list of responses includes all the remaining lea ets sub-images other than the two lea ets obtained from the same entire image as the query.

Finally, we combine the ranking lists obtained from each lea et query obtained from the same original compound leaf, a posteriori, in order to nd the overall responses of the whole image. Di erent state-of-the-art fusion methods are tested and evaluated with regard to the entire compound leaf image query.

This paper is organized as follows. First, we brie y describe some previous work on parts-based plant identi cation and speci cally those that deal with compound leaves. Next, we describe the steps of our lea ets-based retrieval scheme. Experiments and evaluations are presented in the nal section.

RELATED WORK

The elementary analysis and description of leaves, or plants in general, based on their parts are traditionally performed by botanists (mainly using qualitative features) in order to identify species. Some recent computer-vision approaches have used this assumption to enhance plant retrieval results. For instance, the authors of [ 12 ] have combined di erent views of plant organs (such as owers, bark, leaves) using a late fusion process. Analogically, the authors of [ 23 ] have used the same principle (that is the late fusion) to parts of simple leaves. They follow the Manual of Leaf Architecture [ 14 ] for parts de nitions (which divides simple leaves into three parts: the apical, basal and margin parts) and they automatically detect them based on semantic geometric features.

Compound leaf identi cation based on their parts (lea ets) has also been discussed in two previous studies: In the rst one [ 7 ], the authors propose a two-stage compound leaf shape modelling. In the rst stage, it makes the assumption that compound leaves are re ectively symmetric and that their lea ets have the same size and orientation. In this phase, the lea ets are assimilated to uniform circles arranged, pairwise, on either side of the main axis, and dened by their positions, their radius and distance from the main axis. In the second stage, a joint polygonal model is used to estimate the shape of the lea ets (by estimating the length, width, bilateral width, angles of base and apex of each lea et. For each stage, an energy function, based on a color dissimilarity map, is minimized. This method has the advantage that it accomplishes both leaf segmentation and recognition using the same model. However, it is limited by the high computational cost and the intervention of the user to initialize the model's parameters. Moreover, the model's assumptions are so strict that they do not correspond to the reality of the processed data (i.e, they are not valid for all types of compound leaves). In fact, leaves are not always axially symmetric even for pinnately compound leaves (see Figure 3)). Furthermore, the lea ets' size may vary (see Figure 6). The second study that has dealt with compound leaves is presented in [ 2 ]. It looks for the top 3 lea ets, obtained using the following two hypotheses:(1) First, they are located on either side of the main axis (estimated by a polynomial of order 4). (2) they have the most elliptic shapes (de ned by the ratio between the area of the shape and its minimum enclosing ellipse). The nal identi cation stage is based on only one lea et, selected from the three candidates , based on its similarity distance, computed with the complex network shape descriptor, which should be the lowest with regard to the two others.

The approach, presented in this paper, presents similarities with regard to the two rst studies, mentioned above, related to plant organs [ 23 ] and simple leaf parts [ 12 ]. In fact, we aim to identify compound leaves based on an a late fusion of responses of their parts (lea ets) queries. On the other hand, the proposed method presents several di erences from the two last studies [ 2, 7 ], described previously, about lea et-based identi cation: First, we propose to decompose both pinnately and palmately compound leaf shapes, and even lea ets are not similar or symmetric. Second, we use more than one lea et in order to cope with intra-variation of lea ets shapes unlike [ 2 ]. Third, we choose to retrieve each lea et separately and to combine the ranking lists obtained, a posteriori unlike the work presented in [ 7 ], in which an early fusion is rather used. In fact, the early fusion, which consists in concatenating lea et representations in a single one, needs an appropriate lea et representations matching algorithm since lea ets are not selected in the same order from an image to other. 3.

AUTOMATIC LEAFLET EXTRACTION

Parts-based shape decomposition is generally important to shape representation and recognition. Several studies have dealt with this problem. They are mainly based on the perceptual rule of decomposing shapes into regions with important concavities [ 28, 20 ]. The de nition of the best cut that joins minima points is still a challenging issue. Most approaches are based on a recursive procedure or use some optimization criteria, which is a time-consuming task. Domain-speci c knowledge related to shapes may be useful to simplify the decomposition process. In our case, we are dealing with shape of compound leaves (either pinnate or palmate). They are, by de nition, fully subdivided into lea ets, arranged on either side of the rachis (main stalk) in pinnately compound leaves and centred around the base point (the point that joins the blade to the petiole) in palmately compound leaves (see Figure 3) [ 14 ]. From that, we can deduce that the generic perceptual rule can be applied for shapes of compound leaves. In fact, lea et shapes may be seen as regions separated by extra points with deep concavities. In order to determine the points that limit lea ets, we base our solution on the two following botanical assumptions:

In compound leaves, concave points may correspond, besides to lea ets endpoints, to other irregularities such as tooth, lobes, petiole bending or even points derived from the aliasing e ect. These points should be discarded (only points corresponding to lea ets and rachis terminals should be kept). In order to do so, we rst apply a smoothing to the leaf shape. We reject all concave points that are aligned with its two-sided neighbourhood in exion points (see circled green points in Figure 7). The remaining concave points (see Figure 8) are used to determine lea ets. Notice that in exion points and concave points are de ned respectively as the zeros-crossing and the local maxima with negative value of the curvature function of a contour. In practice, we compute the curvature function as presented in [ 21 ].

In practice, we only select, at most, three lea ets in order to try to lead to the compromise between information richness, derived from lea ets variations within the same species (see Figures 4 and 6) and lea ets overlapping problem, that may eventually induce false detections if the number of selected lea ets is high (see Figure 10). In fact, since the processed data (the pl@ntLeaves Scan dataset) hold generally partial lea ets overlapping, the rst lea ets are often approximately complete (see Figure 10). This simple hypothesis (about the number of selected lea ets) ensure a low computational cost, compared to sophisticated methods such as active polygonal models [ 7 ] which also fail to distinguish and delimit overlapped lea ets. Furthermore, trifoliate leaves (see Figure 3) have only three lea ets (see 1st image, 1st column at left in Figure 9). Finally, we judge the relevance of the selected three lea ets, according their size. This is mainly important in the case of approximately totally overlapped lea ets (for example, in the 2nd column of Figure 10, only one lea et is selected). 4.

LEAFLETS TEXTURE DESCRIPTION

We evaluate our multiple lea ets based identi cation approach by testing some texture descriptors described as following. The local description of lea ets texture allow to outline vein networks which are an important attribute in leaf identi cation.

The Fourier histogram (Fourier) proposed in [ 11 ] describes the distribution of the spectral power density within the complex frequency plane. This is expressed using two types of histogram de ned according to two partitions of the Fourier plane: the rst is a disk partition used to di erentiate between low, middle and high frequencies, while the second is based on a partition according to di erent directions of the spectrum. The Edge Orientation Histogram (EOH) [ 10 ], computes the distribution of edge directions. In a leaf, the edges are composed of two parts: the interior and the exterior contours which correspond respectively to the vein networks and the margins.

The Local Edge Orientation Histogram (LEOH) [ 23 ]. Here, instead of accumulating occurrences of gradient orientations in n bins such as in EOH, the LEOH encodes the relative frequency distribution of groups of gradient points contained within a sliding window (blob). All local distributions are combined into a single global histogram.

A Hough histogram (Hough) [ 11 ], is a 2D histogram based on the Hough transform which gives the overall behaviour of pixels in the image along straight lines. Each pixel is represented by the orientation of the gradient and the projection of its position vector onto its tangent vector (i.e the vector orthogonal to the gradient).

MULTIPLE LEAFLETS-BASED QUERIES

FUSION

The proposed multiple lea ets-based fusion method consists in constructing a single overall ranked list by merging, a posteriori, di erent lists obtained for di erent queries of lea ets sub-images.

Let Q be the whole compound leaf image query, and Qi the lea et-based queries where 1 i 3. For the query Q, the visual index is composed of all the remaining images of the database, noted Rn, (where 1 < n < N and N is the number of images in the dataset). In the same way, for each lea et query Qi, the returned images are denoted by Rin and may belong to the set of all the remaining lea et subimages, except the other two lea et sub-images, obtained from the same entire image as the lea et associated to the query Qi. For each image, all queries Qi of lea et sub-images are indexed separately. In order to show the e ectiveness of the lea ets-based fusion, we test three fusion methods:

The leave out method (LO) [18] Responses are inserted in the nal ranking list circularly from di erent lea ets queries lists. The best position of an image among the returned lists is kept.

The inverse rank position method (IRP) [18]

IRP (Q; Rn) =

1 P3 1 1 rRin (Qi) where rRin (Qi) is the rank of the partial response Rin to the partial query Qi. The nal list is obtained by sorting the IRP values in increasing order.

The increasing distance method (DistInc) [27]

The ascending order of the scores, which correspond in our case, to the similarity distance values, de nes the order of the nal list. In fact, we can concatenate the ranking lists obtained by three lea et queries into a single one. We sort the resulted list, in ascending order, in terms of similarity distances in order to obtain the overall ranking list.

Finally, once the overall ranking list is obtained after the late fusion of lea et queries, we apply the knn classi er in order to determine the identity of the query image. 6.

EXPERIMENTAL RESULTS

Experiments were carried out on a subset including compound leaves of the Pl@ntLeaves Scan pictures dataset [ 13 ]. This subset contains 595 images belonging to 16 plant species characterised mainly by a high intra-species variabilities (mainly in terms of lea ets number, size variations, see Figures 5 and 6). The dataset categorisation (into simple and compound) is performed automatically based on the approach proposed in [ 24 ]. We evaluate the e ectiveness of our approach using the correct classi cation rate metric, obtained by the K-NN classi er, for di erent values of k (k 2 f5; 10; 20; 25; 30g). This metric is adequate to the context of plant species identi cation because it re ects the user satisfaction, which is achieved when the accurate species is the mostly present in the k rst responses.

Recall that the principle of our lea ets-based identi cation scheme is that each lea et represents a di erent view of the entire image (i.e, the entire image is replaced by these views in the identi cation scheme). For that reason, we can show the robustness of our approach by comparing its classication results with regard to the classical retrieval scheme based on the global representation of the entire image. We perform several test con gurations using four texture descriptors: Hough, Fourier, Leoh, Eoh (see Section 4), in the description phase, and three fusion algorithms, in the lea ets-based queries fusion phase: IRP, LO, DistInc (see Section 5). Figure 11 presents results for these di erent con gurations. We can see an enhancement in the classi cation rates, obtained by all descriptors and fusion algorithms tested, and for di erent values of k, with regard to the classical retrieval scheme for compound leaves. This prove the e ectiveness of our lea ets-based scheme for enhanced compound leaves identi cation. Also, fusion algorithms IRP and LO perform both the best classi cation rate values. Figure 12 presents the 6 top images, returned for the leoh descriptor, for both the classical retrieval scheme with the entire image (top) and our lea ets-based approach (bottom) using the IRP fusion algorithm, for a specimen that belongs to the species Fraxinus angustifolia. This species was particularly chosen because it holds variability of lea ets number (see some examples in Figure 5). We can see that all the responses returned for the entire image are wrong (they does not belong to the right species). It seems that the leoh descriptor has provided these responses because of the high global similarity in terms of macro-texture. Nevertheless, when we use our lea ets-based approach, we obtain 5 accurate images from the 6 top ones, although that the three last ones have di erent lea ets number with regard to the query image which illustrates well the e ciency of our strategy.

In this paper, we propose a new multiple lea ets-based identi cation approach dedicated to compound leaves. We construct our approach in three phases: (1) The rst is the lea ets extraction. This step is established using simple geometric parameters which are de ned based on botanical observations. We x the number of lea ets to three in order to lead to the compromise of information richness derived from lea ets variations within the same species and false lea et detections induced by lea ets overlapping problem. Our lea ets extraction method has the advantage of being rapid and e cient for di erent types of compound leaves unlike previous methods (2) The second step is the local description of lea ets. In this step, we test four classical texture descriptors (Hough, Fourier, Leoh, Eoh). (3) The third step is the late fusion of ranking lists obtained by each lea ets queries. We test three state-of-the-art fusion algorithms which are IRP, LO and distInc. Experiments were performed on compound leaves of the Pl@ntLeaves Scan pictures dataset for the di erent con gurations of descriptors and fusion algorithms. They have shown an improvement in the classi cation rates with di erent values of the knn classi er, with regard to the classical retrieval scheme obtained by the entire image. Our ongoing work aims at constructing and evaluating the global parts-based leaf identi cation, de ned depending on the leaf type: simple or compound.

References

[1] Enviro . http://www.envirofi.eu/.

[2] A. Arora , A. Gupta , N. Bagmar , S. Mishra , and A. Bhattacharya . A plant identi cation system using shape and morphological features on segmented lea ets: Team iitk , clef 2012 . In CLEF (Online Working Notes/Labs/Workshop), 2012 .

[3] A. R. Backes and O. M. Bruno . Plant leaf identi cation using color and multi-scale fractal dimension . Computer Science , 6134 : 463 { 470 , 2010 .

[4] D. Barthelemy . The pl@ntnet project: A computational plant identi cation and collaborative information system . Technical report, XIII World Forestry Congress , 2009 .

[5] G. Cerutti , V. Antoine , L. Tougne , J. Mille , L. Valet , D. Coquin , and A. Vacavant . Reves participation - tree species classi cation using random forests and botanical features . In Conference and Labs of the Evaluation Forum , 2012 .

[6] G. Cerutti , L. Tougne , D. Coquin , and A. Vacavant . Curvature-Scale-based Contour Understanding for Leaf Margin Shape Recognition and Species Identi cation . In VISAPP , 2013 .

[7] G. Cerutti , L. Tougne , J. Mille , A. Vacavant , and D. Coquin . A model-based approach for compound leaves understanding and identi cation . In IEEE International Conference on Image Processing , 2013 .

[8] J. S. Cope , D. Corney , J. Y. Clark , P. Remagnino , and P. Wilkin . Review: Plant species identi cation using digital morphometrics: A review . Expert Syst. Appl. , pages 7562 { 7573 , 2012 .

[9] J. S. Cope , P. Remagnino , S. Barman , and P. Wilkin . Plant texture classi cation using gabor co-occurrences . In Proceedings of the 6th international conference on Advances in visual computing - Volume Part II , pages 669 { 677 , 2010 .

[10] N. Dalal and B. Triggs . Histograms of oriented gradients for human detection . In Computer Vision and Pattern Recognition , 2005 . CVPR 2005 . IEEE Computer Society Conference on, 2005 .

[11] M. Ferecatu . Image retrieval with active relevance feedback using both visual and keyword-based descriptors . PhD thesis , Universite de Versailles Saint- Quentin-en- Yvelines , 2005 .

[12] H. Go eau, P. Bonnet, J. Barbe , V. Bakic , A. Joly , J.-F. Molino , D. Barthelemy , and N. Boujemaa . Multi-organ plant identi cation . In Proceedings of the 1st ACM International Workshop on Multimedia Analysis for Ecological Data, MAED '12 , 2012 .

[13] H. Go eau, A . Joly, S. Selmi , P. Bonnet , E. Mouysset, and L. Joyeux . Visual-based plant species identi cation from crowdsourced data . In ACM Multimedia , 2011 .

[14] L. A. W. Group .65p. Manual of Leaf Architecture . Department of Paleobiology Smithsonian Institution, Cornell University Press, 1999 - 2000 .

[15] G. A. Haibin , G. Agarwal, H. Ling , D. Jacobs , S. Shirdhonkar , W. J. Kress , R. Russell , P. Belhumeur , A. Dixit , S. Feiner , D. Mahajan , K. Sunkavalli , R. Ramamoorthi , and S. White . First steps toward an electronic eld guide for plants . Taxon , 55 : 597 { 610 , 2006 .

[16] D. J. Hearn . Shape analysis for the automated identi cation of plants from images of leaves . Taxon , pages 934 { 954 , 2009 .

[17] P. JinKyu , H. EenJun, and N. Yunyoung . Utilizing venation features for e cient leaf image retrieval . Journal of Systems and Software , 81 : 71 { 82 , 2008 .

[18] M. JoviA  G, Y. Hatakeyama , F. Dong , and K. Hirota . Image retrieval based on similarity score fusion from feature similarity ranking lists . In Fuzzy Systems and Knowledge Discovery , volume 4223 of Lecture Notes in Computer Science, pages 461 { 470 . Springer Berlin Heidelberg, 2006 .

[19] N. Kumar , P. Belhumeur , A. Biswas , D. Jacobs , W. Kress , I. Lopez , and J. Soares . Leafsnap: A computer vision system for automatic plant species identi cation . In Computer Vision a^AS ECCV 2012, Lecture Notes in Computer Science , pages 502 { 516 . 2012 .

[20] H. Liu , W. Liu, and L. Latecki . Convex shape decomposition . In Computer Vision and Pattern Recognition (CVPR) , 2010 IEEE Conference on, pages 97{104 , 2010 .

[21] F. Mokhtarian and A. Mackworth . Scale-based description and recognition of planar curves and two-dimensional shapes . IEEE Transactions on Pattern Analysis and Machine Intelligence , PAMI- 8 : 34 { 43 , 1986 .

[22] F. Mokhtarian and A. K. Mackworth . A theory of multiscale, curvature-based shape representation for planar curves . IEEE Transactions on Pattern Analysis and Machine Intelligence , 14 : 789 { 805 , 1992 .

[23] O. Mzoughi , I. Yahiaoui, N. Boujemaa , and E. Zagrouba . Advanced tree species identi cation using multiple leaf parts image queries . In IEEE International Conference on Image Processing (ICIP) , 2013 .

[24] O. Mzoughi , I. Yahiaoui, N. Boujemaa , and E. Zagrouba. Automated semantic leaf image categorization by geometric analysis . In ICME , 2013 .

[25] Y. Nam , E. Hwang, and D. Kim . Clover: A mobile content-based leaf image retrieval system . In Digital Libraries: Implementing Strategies and Sharing Experiences, Lecture Notes in Computer Science , pages 139 { 148 . 2005 .

[26] A. R. Sfar , N. Boujemaa , and D. Geman . Vantage feature frames for ne-grained categorization . In CVPR. IEEE , 2013 .

[27] R. Snelick , M. Indovina , J. Yen , and A. Mink . Multimodal biometrics: Issues in design and testing . In Proceedings of the 5th International Conference on Multimodal Interfaces , ICMI '03 , 2003 .

[28] L. Wan . Parts-based 2d shape decomposition by convex hull . In Shape Modeling and Applications , 2009 . SMI 2009 . IEEE International Conference on, pages 89 { 95 , 2009 .

[29] Z. Wang , Z. Chi , and D. Feng . Shape based leaf image retrieval . VISP , pages 34 { 43 , 2003 .

[30] X. F. Wanga , D. S. Huanga , J. X. Dua , H. Xua , and L. Heutted . Classi cation of plant leaf images with complicated background . AMC , pages 916 { 926 , 2008 .

[31] N. Yunyoung , H. Eenjun , and K. Dongyoon . A similarity-based leaf image retrieval scheme: Joining shape and venation features . Computer Vision and Image Understanding, 110 : 245 { 259 , 2008 .