This article presents the participation of the ReVeS project to the ImageCLEF 2013 Plant Identi cation challenge. Our primary target being tree leaves, some extra e ort had to be done this year to process images containing other plant organs. The proposed method tries to bene t from the presence of multiple sources of information for a same individual through the introduction of a late fusion system based on the decisions of classi ers for the di erent modalities. It also presents a way to incorporate the geographical information in the determination of the species by estimating their plausibility at the considered location. While maintaining its performance on leaf images (ranking 3rd on natural images and 4th on plain backgrounds) our team performed honorably on the brand new modalities with a 6th position.
Considering the identi cation of a plant species, the study of its leaves is a
common path, and undoubtedly the most tting for an automatic system, given the
relative visual stability of the planar, slowly growing objects that are leaves.
However, they are not the organs on which was built the classi cation of the Plantae
kingdom, which considers the temporal proximity of the species as a baseline
grouping criterion. Flowers and fruits, representative of the plant's intrinsic
reproduction properties are the most evident witnesses of this proximity, and have
been used since the earliest classi cation works [
The main objective of the ReVeS project is to build a tree species identi
cation system based on an explicit botanical description of leaf images. Unlike for
the previous editions, the challenge laid by this year's identi cation task [
This year's database approximately doubled in size compared to last year, and
it was separated into 2 big categories :
{ Sheet As Background images, containing only leaves, and thus
combining the categories Scan and Pseudoscan of previous editions [
Among the 20985 images forming the Train database, we could nd 250 plant species, but a distinction can be made between tree species (127 generally ligneous, taller than 1.2 m plant species) that were the ones considered in the previous editions and the ones represented in the SheetAsBackground category, and the 123 rather herbaceous remaining species. This distinction is important for the identi cation, the tree species presenting for example bark in the Stem images, that may be analyzed in a speci c way, more accurately than a generic stem image.
The Test database contained 5092 images with the same information on the content, but with no indication whatsoever whether the species was a tree or not, which forced us to make assumptions and selections for the good running of our methods. 2.1
Manual Image Formatting Before any actual processing, we had to perform some transformations on the Natural Backgroung images to convert them into suitable entry points for our algorithm. As mentioned previously, our objective is to develop a mobile application for identi cation, which allows a certain part of interaction with the user, by giving guidelines for the picture taking or directly asking for help in the image processing phase though the tactile interface. The procedure we applied on the images simply simulates what we would have asked of the supposed user of the application.
Most of the Leaf, Fruit and Flower images contained more than one example of the desired object. A rst selection was done by placing all the images for which no object big and in focus enough could be found in the Entire category, and keeping the others. We subsequently cut those images to a region of interest containing only one object, and rotated them (90 or 180 rotations only) so that the resulting images correspond to what input a user asked to take a photograph of one, vertically oriented, organ would have given. A typical example of this process can be found on the Figure 1 (b).
In addition to this rst manipulation, we drew on top of each of the obtained images a rough mark inside the object that is used as an initialization of our segmentation algorithms. Note that in the case of leaves where the methods are the most developed, a distinction is made between simple and compound leaves : in the rst case, one single mark is required whereas we require three distinct marks in three of the top lea ets for second type of leaves. An example of such quick colouring, as it would be done with the nger on a mobile device, is shown in Figure 1 (c).
For Stem images, where no segmentation process was considered, we simply rotated and cut the images to obtain a vertical view containing only bark pixels. The images clearly corresponding to non-tree species were discarded to the Entire category, as well as the non-tree leaves and fruits. Unfortunately, we did not have the time to do anything with the images of this extended category. 2.2
Leaf Understanding
Leaf images were processed with almost the same method as last year [
The main di erences concern the processing of compound leaves in which we
try to describe all the lea ets at the same time. This is achieved by estimating
the number and location of the lea ets, and jointly modelling their global shape,
using highly variable deformable templates [
Another improvement lies in the representation of the margin that now uses a string-like structure to keep track of the spatial repartition of teeth along the leaf contour. When our former descriptors used aggregative methods (computation of histograms, or averaging of curvature-scale measures) this new description is more discriminant as it can represent di erent levels of dentition or di erent orderings of teeth. 2.3
Fruit, Bark and Flower Image Analysis The methods we used on the other organs were not nearly as dedicated and thought through as the ones elaborated for leaves, as they are not our main concern. We made the decision to rely on statistical shape descriptors and on vocabulary learning to produce color, texture and interest point based features. The only particularity came from the fact that in the case of Flower and Fruit images, a segmentation had to be performed rst for these descriptors to be extracted correctly.
Fruits and owers may be easier to isolate in an image given how their
colors stand out, but they still require some advanced processing [
To represent the color information, we used a bag-of-colors-like feature based
on an organ-speci c color vocabulary, much like what can be found in other
works considering owers [
The same idea was adopted for texture description keeping for each pixel in
the region, instead of a 3-channel L*a*b* vector, a vector of 30 values
corresponding to the responses obtained by ltering the image with the commonly
employed Gabor wavelets [
Finally, we computed the classical bag of visual words by repeating the same procedure again, this time on SURF descriptors obtained on detected keypoints (limited to 500 per image, for computational reasons). The only di erence is that the histograms are not normalized by the number of keypoints extracted on the image, otherwise it is in every respect similar. These bag of words were used on Stem images only, with a vocabulary of 1000 visual words.
To capture the shape properties of the segmented organs, we decided to rely on well established, robust shape descriptors, commonly used in the literature for object recognition purposes, namely : { Centered moments (8 values) { Eccentricity (1 value) { Hu moments (7 values) { Zernike moments (9 values)
These values were computed on the binary segmentation images resulting from the algorithm described above. Therefore, they were only extracted on Fruit and Flower images. 3
We chose to view the problem of species identi cation as a classi cation
problem rather than an image retrieval one, and subsequently trained classi ers to
perform the recognition. Unfortunately, the time dedicated to this task was too
short to reproduce the random forest classi cation that gave us the best results
last year [
Naive Distance-based Classi cation For each set of the aforementioned descriptors, and for each considered species, we were able to model the distribution of the parameters by gaussians, evaluating the means and standard deviations from all the examples. This was also what we did for the histogram-like features that are maybe not the most suitable for this type of approach. The only di erence concerns the strings describing leaf margins, non-vectorial structures that had to be addressed di erently, simply by estimating an average string per species.
The set of considered species, on which the training was performed and that are likely to appear in the classi cation results, was also di erent depending on the organ : { Leaf (L) : 126 tree species, separated in simple-leaved and compound-leaved { Fruit (F ) : 44 tree species with enough examples { Flower (f ) : 139 species, essentially herbaceous { Bark (B) : 78 species, only ligneous { Entire plant (p) : No training, by lack of time
To classify a new image, we simply evaluate the distances do(x; S) of the descriptors extracted on it to the gaussian models of the retained species (So) for the corresponding organ o. The metric we used for histogram-like features is a simple L2 distance, for the string structures, a normalized Levenshtein, and for the vector features a so-called ellipsoid distance, designed to take variability into account, without distorting the parameter space as would the Mahalanobis distance. That distance represents the distance of a point x to the surface of the ellipsoid de ned by the center and the axes of the covariance matrix . Using the Mahalanobis distance kx kM = (x ) 1(x )T , the metric we compute can be written as : kx kE = kx k2 max 1 kx 1 kM ; 0 (1)
Each one of the ND sets of descriptors extracted on the image thus produces a distance, that is then normalized by the average distance to the correct class (learned for each feature during the training phase). Once those terms are on the same scale, they are just summed, and the set of all considered species are ordered according to this last measure. This way, for each image, we produce a ranked list of species along with a distance value measuring their similarity.
Individual-based Information Fusion This ranking could be enough to give a suitable answer per image, but it is much more interesting to try and make use of the di erent modalities available for a same individual plant. The images corresponding to a same value of the IndividualPlantID are actually di erent views of the same plant, taken at the exact same location. Consequently, the answer to each of these images needs to be the same, and every element that has to be taken into account to produce it with as much accuracy as possible.
To perform an e cient fusion of these various data sources, we used a weighted sum of con dence measures obtained from the distances to species issuing from the di erent classi ers. The fusion process is illustrated in Figure 5. An individual may be associated to a certain number of images (up to 24) of di erent organs. Each one having been processed independently by the according classier, we retrieve for each of the no images of the organ o the distances do (1::no) to the di erent species considered in the context of this organ.
The distances are rst converted into con dence measures. Each distance value being a sum of ND;o terms expected to be equal to 1 for the correct species, we de ne the con dence score associated to the distance as following : Co(x; S) = 8 >< exp >: 0 do(x; S) 2
ND;o if S 2 (So) if S 2= (So) (2)
In the end, even if distances were computed for only a subset of all the
considered species, we now have for each image a set of con dences for each
and every species. The value is 0 for those that were not learned but they are
still present in the possible answer list. These con dence values all lie between
0 and 1 and the normalization of the distance by nD;o makes sure that they
constitute comparable measures even for di erent organs. However, the results
we obtained on the di erent modalities tended to show that some are more
reliable than others. With this in mind, the con dence measures are weighted
by an organ-wise coe cient wo derived from the classi cation rates we could
observe on the training base, in a common attempt of giving more importance
to reliable sources of information [
Finally, we used an additive process on these weighted con dences, in order not to penalize too much the species that were absent for a given organ. The sum of weighted con dences is then normalized again by the sum of weights to remain between 0 and 1. The nal con dence score for a species S is then computed as :
C(S) =
Po2fL;f;F;Bg (Pin=o1 wo Co(xi; S))
Po2fL;f;F;Bg (Pin=o1 wo) (3)
With these similarity measures, we can produce a new ranking of the species, this time all the 250 of them, that will be the same for all the images of the considered individual. The information available on each of the image is then taken into account, with a degree of trust depending on the modality, and the ordering that would have been obtained on one image only is then re ned and hopefully improved by the content of the other associated images. Of course in the case when there is only one image per individual, the order remains unchanged, but when di erent organs are available, or di erent leaves of the same plant, the species appearing regularly as close will be put up front. 3.3
Introducing Biogeographical Knowledge Geographical and environmental factors play a big part in the development (and survival) of plants, and not all species are likely to be found under every conditions. Knowing where a plant was observed, and what species may grow in that particular location might constitute a great help for identi cation. With this perspective, it was a core objective of our project to come up with methods to bene t from the location (easily accessible through GPS on mobile devices, for example) to enhance the identi cation by knowing in advance the plausibility of species.
In this respect, a large-scale work of digital cartography was undertaken
(focusing on the French metropolitan territory) to map some environmental
parameters that are related to the presence of species, with a resolution of 90 meters.
Indeed, variables such as temperature, altitude, precipitation, humidity,
exposition have a key in uence on the repartition of species. Along with these raw
explanatory variable, we also considered more global, integrative values
elaborated by botanists and phytosociologists :
{ Altitudinal zonation, or vegetation levels as de ned in phytogeography [
The images in the PlantView database were all referenced in terms of longitude and latitude, and most of them issued from the french territory, making them suitable to experiment this geographical approach. Unfortunately, there was a big incertitude about the precision of this location, whether it was the place where the image was actually acquired or simply uploaded. In the case of scanned leaves, is it the location of the tree or the scanner that is supplied? Therefore, trying to include local parameters like the raw measurements hardly made any sense, and we chose to take only levels and regions into account. The maps for those two variables can be seen in Figure 6.
To derive the plausibility of nding a species at a given location, we did
not use the training base but included external botanical knowledge about the
species. Based on the repartition knowledge and maps one can nd in oras
[
Then, for given values of the region index g and of the vegetation level l, it is possible to compute a numerical value of plausibility for each species given its theoretical repartition. Knowing the presence values of the species in the region p(S; g) and the level p(S; l) we simply derive an additional plausibility score : P (S; g; l) = 8 1 if p(S; g) = 5 > >>> 0:95 if p(S; g) = 4 > > 1 <>> 0:9 if p(S; g) = 3 2 > 0:8 if p(S; g) = 2 > >>> 0:7 if p(S; g) = 1 > > >: 0:5 if p(S; g) = 0 + 1 ( 1 if p(S; l) = 1 2 0:5 if p(S; l) = 0 (4)
For visualisation purposes, we computed this plausibility for di erent species all over the French territory, for which the values of vegetation levels and ecological regions were mapped. The result can be seen as a repartition map of the species, in which the places where the species is most plausible appear in darker green, on those where it will certainly not be found in lighter green. Examples of such maps are shown in Figure 7.
For the species on which we had no prior repartition knowledge, we ensured the geography would not have a big in uence on classi cation by setting the P (S; l; g) to 1 for all values of l and g. This of course is not an optimal solution, and actually favours those species, since they will at least be as plausible as the other species, whatever the location.
The way we insert the geographical knowledge into the classi cation process is fairly simple. Given the coordinates (lon; lat) of the observation, we can extract from the mapped data the values g(lon; lat) and l(lon; lat) of the corresponding GRECO and level. Then for each species S the con dence score used to rank it simply becomes :
CG (S; lon; lat) = P (S; g(lon; lat); l(lon; lat)) C(S) (5)
The introduction of this geographical plausibility leaves unchanged the condence score for the species that are perfectly adequate with the observation's location. However, it will penalize the species which are not, making them drop in the rankings and conversely letting the more plausible species make their way up to the rst answers. It is a way to make a rearrangement of the top answers without actually overshadowing the information extracted from the image. It is in any case coherent with the idea that geography should not contradict the evidence found in the content of the image, but simply provide a lighting on the results of its analysis to make a better nal decision. 4 4.1
Training Results The classi cations of the di erent plant organs do not perform all the same, the most e ective one naturally being the classi cation of leaves on plain backgrounds. As a matter of fact, leaves are the organ for which most work has been dedicated, and the methods used to process them were the most extensively tested. It is then only natural that the experimental results makes them appear clearly as the best performing modality.
The Figure 8 sums up the experimental classi cation rates obtained for the 4 modalities we addressed. Those scores were measured on the Train database, in a cross validation process (validation rates are displayed, with the rates obtained on the training data left in lighter color). If the Leaf category (126 classes, 4 descriptor sets) clearly bene ts from the e orts put in the segmentation and the botany-inspired description (nearly 85% of presence of the correct species in the top 5 answers, up to 70% in the case of natural images), the other modalities are not as far behind as expected.
The Fruit category (44 classes, 3 descriptor sets) is the second best performing (more than 50% at 5 answers) but this performance has to be moderated by the fact the the number of classes is relatively small. On the other hand the Stem category (78 classes, 2 descriptor sets) performs honorably (more than 40% at 5 answers) given the number of species and the rawness of the description used. Same goes for the Flower modality (139 classes, 2 descriptors) for which it might be interesting to add that most of the accuracy (37% at 5 answers) actually comes from the simple color feature we used. All in all, these results underline the limits of the descriptions (hastily) implemented, and in the case of fruits and owers the di culties of the segmentation, given the low contribution of the shape description. It is also crucial to point out the greater abstraction power of the dedicated description created for leaves compared to the generic features used on the other organs ; this is visible in the blatant gap between the curves obtained on the training and validation sets.
A very interesting point is to measure the contribution of geography in the performance of the classi cation. To do so, we considered only leaf images from tree species that are native in France, and on which we had the necessary theoretical geogaphical information. The Figure 9 shows the comparison between a classi cation where the geographical plausibility of the species is taken into account and one where it is not. More than the actual rates we obtain, it is clearly appearing that geographical knowledge leads to a better classi cation, and simply improves the ranking of the correct species. The Figure 10 shows this improvement by measuring the di erence between the rank of the correct species in the classi cation without geographical information and with it, a difference that goes in the right direction for a major share of examples for which the ranking could still be improved. Even with unprecise location (obviously for scanned images) the region and level prove to give enough information to re ne the decision in a more accurate way.
(a) We submitted two distinct runs this year, both relying on the same visual analysis process but di ering in the fusion phase. They both make use of the IndidualPlantID eld, but the second run (LirisReVeS run2) additionally used the longitude and latitude coordinates to compute the geographical plausibility of the species and use it to rearrange the results of the image processing phase.
Both runs were then mixed (visual + textual) runs, with an automatic method on the SheetAsBackground category and manual initalizing on the NaturalBackground category. To be exhaustive, the images remaining untreated, and not belonging to a processed individual were submitted as specimens of Ginkgo biloba, L. with an absolute con dence of 1.
As expected, the SheetAsBackground category, containing almost only leaves, is the one on which our leaf-focused algorithms gave the most satisfying results. As shown in Figure 11, we reach a score of 41% bringing us in 4th place among the 12 teams that have submitted runs.
Concerning the NaturalBackground images, the results are with no surprise lower, but our ranking on the Stem images (3rd place, yet with a score of 16%) was unexpectedly quite interesting. Our low performance on fruits and owers (respectively 8% and 10%) is a sign, if needed, that the representation of such organs requires more dedicated descriptors, and would probably bene t from the introduction of some kind of model to focus on the discriminant criteria from a botanical point of view. And nally, our 3rd place on the Leaf images corresponds to our previous rankings, with again the same limitations (only broad-leaved tree species considered, some images out of our processing scope, learning performed on plain background images) that might explain the rather low raw score value of nearly 17 %. Fig. 12. Results on the NaturalBackground images : 3rd on Leaf images (a), 6th on Fruit images (b), 3rd on Stem images (c) and 7th on Flower images (d)
Concerning geography, the run LirisReVeS run2 gave slightly better results than the run LirisReVeS run1 from which the location information was completely left out. But the di erence (almost 1% in every category) does not appear as clearly as it did on native species, and it is hard to draw any conclusion. There were however too many manifest aws in the context of this task to make this a valid experiment for us. First, we had theoretical phytogeographical data at hand for hardly half of the 250 species, which made the plausibility values we computed only half-relevant. More importantly, because of the doubt about the precise location of the observation, we were unable to use the local geographical parameters that might be more accurate. The fact that many images come from cities, where it is more di cult to assess whether the plant grew in its natural environment, also made the problem a bit less clear. It is nevertheless an encouraging step towards the use of geographical information to help the identi cation, a direction that will in any case be further investigated. 5
This year's task constituted a real new challenge for our leaf-oriented project, and it came as no surprise that the generic, mechanical, methods we had the time to implement on fruits, barks and owers were largely outperformed by the now mature leaf analysis process put into practice in the Folia applicationx. It was however a good chance to experiment with a late fusion mechanism which seemed to bear fruit. It was also the opportunity for a rst testing of the novel approach proposed to combine the image information with geographical knowledge. The results of this last contribution are engaging, and we believe that with an adequate scope of species and a greater certainty on location it may constitute a real asset for eld plant identi cation.
We would like to thank Tristan Coulange for his contribution in the hard task of developing of competitive ower segmentation methods, as well as Marine Aghadjanian, Nicolas Charrel, Nadya Dahir and Maxime Melinon for their precious help in processing and analyzing thousands of fruit, bark and ower images. x
: https://itunes.apple.com/app/folia/id547650203