This paper describes a citizen science system for ora monitoring that employs a concept of missions, as well as an automatic approach for ower species classi cation. The proposed method is fast and suitable for use in mobile devices, as means to achieve and maintain high user engagement. Besides providing a web-based interface for visualization, the system allows the volunteers to use their smartphones as powerful sensors for collecting biodiversity data in a fast and easy way. The classi cation accuracy is increased by a preliminary segmentation step that requires simple user interaction, using a modi ed version of the GrabCut algorithm. The proposed classi cation method obtains good performance and accuracy, by combining traditional color and texture features together with carefully designed features, including a robust shape descriptor to capture ne morphological structures of the objects to be classi ed. A novel weighting technique assigns di erent costs to each feature, taking into account the inter-class and intra-class variation between the considered species. The method is tested on the popular Oxford Flower Dataset, containing 102 categories and we achieve state-of-theart accuracy while proposing a more e cient approach than previous methods described in the literature.
Citizen science is not a new concept: the idea of conducting research by citizens, gathering crowdsourced data that is analyzed for scienti c purposes, has been active for generations, dating back to 1900, when the Christmas Bird Count (CBC)1 project started. It was a form of mass collaboration citizen science project that used paper forms sent by post to the responsible society or research group of scientists who requested the data.
Nevertheless, it is noticeable that the power of crowdsourced science is growing intensely nowadays and such projects are becoming increasingly popular, even gaining the attention of major news media. For instance, projects such as eBird2 and Galaxy Zoo3 are able to engage hundreds of thousands of volunteers and are being broadly used for scienti c and educational purposes.
The citizen science's popularity boost is given by the fact that data collection and analysis tasks became much easier to address. Even simple and low cost smartphones are equipped with GPS and high-resolution cameras that allow huge amounts of data to be collected with small e ort. Also, the Internet brings together a high number of volunteers to work on this data remotely and simultaneously. However, there are still di culties that prevent the further growth of this methodology.
One challenge of citizen science projects is how to obtain
high and constant engagement of the users. Although some
volunteers are motivated by the scienti c contribution on
its own, it is possible to resort to \gami cation", the use
of game elements in non-game contexts [
Project Noah4, for instance, is a large scale project in
which every report is assigned to a speci c mission, whose
goal is to monitor certain types of ora or fauna classes in
a speci c location. There are also many aspects that can
increase user motivation, as observed in [
This last point highlights an additional drawback of proje1http://birds.audubon.org/christmas-bird-count 2http://www.ebird.org/ 3http://www.galaxyzoo.org/ 4http://www.projectnoah.org/ cts monitoring biodiversity: relying on the users having the knowledge to classify, without further assistance, the specimens being reported. This can be a di cult task when one considers the large number of species that share similar morphology. Misclassi ed inputs can compromise the environmental research, and the impossibility to assure the quality of the analysis made by non-experts may be the bottleneck of such projects. In this regard, an algorithm to assist the user would be very useful. A system that could automatically recognize an input image with small e ort, or at least retrieve a list of the most similar species, could make the user feel more con dent, perhaps even engaging a larger number of volunteers, preventing errors and accelerating the identi cation process. If such a system could be deployed in a mobile device, the bene ts would be even greater, once the classi cation could be executed at collection time.
To assist non-expert classi cation, automatic works in the
ora eld { which is the domain of focus of this paper {
are getting good attention, and recent studies were able to
achieve good results in leaf and ower classi cation [
Besides robustness, automatic or semi-automatic methods
can reduce data collection time. For instance, as indicated
by Zou and Nagy [
1.1.1
Regarding automatic classi cation, the LeafSnap project5
o ers a good di erential with respect to other citizen science
projects, since it runs a shape-based algorithm to classify
plant species from their leaves automatically [
The segmentation method proposed by LeafSnap might be considered harmful in an environmental aspect, once it requires the extraction of the leaves from their natural surroundings. Also, lighting variations can a ect the background color and interfere in the segmentation result. Finally, it is hard to extend LeafSnap for additional domains other than leaf classi cation: besides taking shape information as the only feature for classi cation, a segmentation method based on a xed background is only feasible for static species, once it is clear that, when photographing specimens that can move, there is no guarantee that they will remain in place.
Table 1 shows a list of various citizen science projects in the botanical eld. To the best of our knowledge, these are the most representative projects in this area. Their limitations highlights the previously discussed issues regarding the di culty of species identi cation or adaptation of the volunteers routine, once the data upload requires a computer and/or Internet connection.
Our goal is to develop a system similar to LeafSnap, but focusing on owers, and replacing the user e ort for segmentation by a simpler and less intrusive action taken in his own device. Since our study is part of the citizen science context, user interaction is accepted, once we assume there is always a volunteer operating the system in his smartphone or tablet. Our platform is developed with the concern of be5http://leafsnap.com/ ing attractive to users so it can reach the largest number of volunteers as possible. Therefore, our ower classi cation algorithm must have high accuracy and also must be fast enough so that it can run in the web browser and also in mobile devices. Besides, we use missions to engage and entertain the volunteers, following gami cation principles. In order to compare our classi cation results with a benchmark, we use the challenging Oxford Flower Dataset6. 1.1.2
In the ower classi cation eld, previous studies worked
on small datasets of 10 to 17 ower species [
Using larger datasets, consisting of 30 to 79 ower species,
accuracy rates vary from 63% - 94%, combining a variety
of approaches such as histograms in HSV color space,
contour features, co-occurrence matrix (GLCM) for texture
features, RBF and probabilistic classi cation [
Works on the Oxford Flower Dataset, that contains 102
ower species, reported accuracy rates no greater than 80%.
The dataset was introduced by Nilsback and Zisserman[
(a) Spear Thistle (left) and (b) English Marigold (left) Artichoke (right). and Barbeton Daisy (right). (c) Spring Crocus.
(d) Bougainvillea.
Using the Oxford Dataset, studies working with
unsegmented images demonstrate lower accuracy in classi cation
and require powerful classi ers, that are computationally
expensive [
Using segmented images, the authors are able to achieve
a more robust classi cation for this dataset, though none of
the considered works is suitable for real time. Nilsback [
Our main goal is to develop a citizen science application for ower monitoring whose collection is oriented by missions. Also, to assist the volunteers, we propose an algorithm for classi cation of ower species that meets the following requirements:
(i) Has high accuracy: We propose a novel approach that
relies on e cient histogram-based feature descriptors that
capture both global properties and ne shape information of
objects. This is made while leveraging a learning-based
distance measure to properly weight the feature contributions,
which is a critical step for increasing accuracy as
demonstrated in previous studies [
(ii) Is fast: Mobile-based applications have a challenge regarding connectivity: users can only submit requests and receive answers if their devices have access to the Internet. There are several locations in which this kind of service is not a ordable or has poor quality, making image transference prohibitive. Our algorithm is fast enough to run directly in the devices (without transferring information to a server), so the user does not have to rely upon a network connection and wait too long for a response, considering additional time latency for uploading the image and retrieving the classi cation results.
This paper is structured as follows. In Section 2, we describe the overall structure of our citizen science platform and, in Section 3, we describe the core approach for negrained classi cation. In Section 4, we show and discuss the obtained classi cation results on the considered dataset and our conclusions and future work are described in Section 5.
The system is comprised of two main parts: a web portal, organized toward data visualization and community organization, and a mobile application, aimed at data collection. Both components arrange user-contributed data around the concept of missions, which aggregate users with the common goal of collecting structured and unstructured data about a certain class of observations.
The collected data is structured in a set of attributes, whose pre-established values are lled in by the user and refer to the domain being registered. This structure allows us to provide query-by-example features, which helps to avoid common input errors and establish a common vocabulary for user-contributed data. The unstructured data, on the other hand, is comprised of images only, since we are dealing with plants. (a) Image upload (b) Free-hand user marker (c) Classi cation of the (in white) segmented input (d) Other results (e) Screen for geo-analysis
Missions are created by users and can be set as public or private, as determined by the mission's owner, since some of them may contain sensitive geo-spatial data about observations belonging to endangered species. As the number of missions in the system may be rather large, the portal ranks them according to their activity, to make it easier for users to engage with the rest of the community.
It is currently possible to collect geo-referenced photos with consumer-grade mobile devices, using their cameras and GPS receivers, as mentioned in Section 1. The steps the user has to go through in order to submit a contribution are highlighted in Fig. 2. For the purpose of submitting a report, the user rst needs to join one mission. After the picture is taken, he must loosely delineate the region of interest in the image. This is used for a segmentation process that will be described int the next Section. This step can require user con rmation for proceeding with the next task, which is the automatic classi cation of the segmented input. The system retrieves the top 5 matches that are more similar to the uploaded image and the user can select the correct one or browse within a list of all the registered classes.
In situations where there is no connectivity, the algorithms can be executed directly on the devices, as a signi cant range of current mobile processors have multiple cores and some of them have vector processing instructions, which are already supported by popular libraries for mobile platforms, such as OpenCV. After the report is submitted (and synchronized with the server, in situations of intermittent or no connectivity), other users can view and bookmark this report. The feedback is provided to the users as bookmarked reports. Additionally, the user may ll in a form containing values for the observable raw morphological attributes of the sample. The set of supported attributes is de ned by the mission's owner at the moment it is created, where he can de ne the required elds according to the mission's needs. For the botanic eld, some examples of attributes are stem diameter, leaf thickness and so on.
In order to obtain more accurate classi cation results, we let the user choose the category of the image he is uploading. We currently only have trained classi ers for owers, but the overall concept of the system is generic for accepting data belonging to other categories. Since observations can be arbitrary, we need to load the corresponding classi er that was trained with the relevant data for the current category. 3.
This section describes the proposed algorithm for ower recognition that is divided in segmentation and classi cation steps. Both tasks are designed to be fast and precise and are generic enough to be used for classifying categories other than owers. 3.1
We use the GrabCut algorithm [
A Gaussian Mixture Model is used to learn the pixel distribution in the background (outside the marker) and possible foreground regions and a graph is built from this pixel distribution. Edges' weights are given according to pixel similarity (a large di erence in pixel color generates a low weight). Finally, graph and image are segmented by a mincut algorithm. See an example in Fig. 2(b) of a user marker and corresponding segmented output in Fig. 2(c).
The advantages of this approach are its simplicity in a user perspective, generation of good segmentations and that it is faster than the segmentation approaches proposed in Section 1.1.2, requiring average time of 1.5 second. 3.2
We compare multiple features (i.e. color, texture and shape-based features) with histogram matching, which is fast and invariant to rotation, scale and partial occlusion.
For each class, a weight is assigned to each feature and (a) Two classes that have similar color for (b) Species that can have many di erent col- (c) Two classes with similar shape and color, all samples, but variations in shape. ors, while having a similar shape. but with large variation in texture. the segmented images are matched by comparing their histograms, using a kNN classi er. The di erence between two histograms is computed by a metric based on the Bhattacharyya distance that is described as follows.
Consider histograms H1 and H2, with n bins. Let H1(i) denote the ith bin element of H1, i 2 1 : : : n. H2(i) is de ned analogously. The distance between H1 and H2 is measured as: v u d(H1; H2) = uu1 u t
Pin=1 pH1(i) s n
P H1(i) i=1
H2(i) n P H2(i) i=1
Note that low scores indicate best matches. Our algorithm compares histograms built on three features cues. 3.2.1
We build a histogram of the segmented image's colors in the HSV space, which is known to be less sensitive to lighting variations, using 30 bins for hue and 32 bins for saturation. Image quantization is applied before computation and the histograms are normalized in order to be comparable with the proposed distance metric.
The texture operator applied in the segmented image is
LBP (Local Binary Pattern) [
We use the extended version of LBP [
We propose the use of two simple and fast descriptors that, when combined, are able to represent di erent shape characteristics. The shape contour is partitioned in 72 bins using xed angles, resulting of m contour points. For each point pi, i 2 1 : : : m, we compute modular and angular descriptors, and both vectors are later represented as separate histograms.
The modular descriptor is taken by computing the point distance from the shape centroid, normalized by the major distance dmax computed in this process. It measures the contour relation according to the shape mass center, and elongated shapes are easily distinguished from round shapes. The angular descriptor computes the angles between p and its neighbors pi 1 and pi+1, measuring the smoothness of the contour. The proposed descriptor becomes powerful by merging two simple features being able to represent, for instance, petals length, density and symmetry. 3.2.2
In order to achieve higher accuracy, we assign di erent weights to the features when matching each class, as to contemplate the situations described in Fig. 3. Our idea is to
nd which features are more discriminative taking into account each feature variation inside a same class, but also, with respect to the global variation of all classes. We learn, for each class, one weight for each of the four feature descriptors. We estimate few weights due to the small number of training samples per class.
We consider N classes, each containing M training samples, evaluated according to P features7.
Let Hip;j denote, for a class i, the histogram of a sample j regarding feature p. We compute the mean histogram of a feature p for a class i as:
M
P Hip;j Hip = j=1
M
Likewise, we de ne "ip as the mean distance per class i, regarding feature p, computing the distance from all the training samples to the mean histogram. This help us to evaluate intra-class variations, once we are considering the di erence between all samples with respect to a histogram that estimates the overall structure of the class.
Also, we compute the mean distance per feature p ("p) as follows:
M
P d(Hip;j ; Hip) "ip = j=1
M N P "p
i "p = i=1
N and we use this information to estimate inter-class variations between all types of species regarding each considered feature. Finally, the weight attributed to each of the considered features p in class i is given by: 7In our tests, we use N = 102, M = 20 and P = 4, since we consider two histograms for shape information.
(a) Input belonging to Bearded Iris (BI) class, with large variation in color. (b) Input belonging to Bishop of Llanda (BoL) class, with small variation in color. (c) Input belonging to Spring Crocus (SC) class, with large variation on texture. (d) Input belonging to Blackberry Lily (BL) class, with small variation in texture. and we are able to take into account intra-class and interclass variations between all training samples, estimating how each feature should be considered when evaluating each considered class.
After the weights computation, we employ a kNN classi er in the test phase where the cost of matching an image I with an image IC of class C is computed as:
C(I; IC ) = XP (C; p) p=1
d(HI ; HIC ) maxi("ip) and select the classes with the lowest costs. We choose the kNN classi er because it is robust and performs well with training sets whose dimension is similar to the ones we are dealing with.
Our tests followed the speci cations for using the Oxford Dataset as a benchmark, considering 20 training samples per class and computing the mean-per-class accuracy. In this dataset, the number of images in each class varies from 40 to 258. Fig. 4 shows the top 5 results for 4 di erent inputs. Every image is segmented as described in Section 3.1 and the used weights are given as described in our metric learning method. Note how the weights a ect the classi cation results.
Table 2 describes the algorithm's accuracy for a variable number of top matches, with and without using metric learning. Once our platform returns the top 5 most likely classes, we can see that the system is able to achieve a very high accuracy rate for a very challenging dataset.
E ciency can not be compared precisely, because this
information is not reported in all the previous works. However,
our average time for extracting and matching all features
proved to be 4 times faster than running SIFT in the
images' foreground region. The baseline work of Angelova and
Shenghuo [
Finally, we replicated the training instances, and Table 4 shows that our method is still e cient for much larger training sets. Approximate nearest-neighbor methods based on hashing or kd-trees could also be used for obtaining even higher e ciency.
This paper proposes a citizen science platform based on a novel approach for ower recognition. We introduce a strategy for comparing feature histograms for ne-grained classi cation, a robust shape descriptor and a metric learning approach that employs di erent weights to each feature, that can improve classi cation accuracy signi cantly. Our algorithm is extremely fast, being suitable for o ine mobile applications and was able to outperform previous works using the popular Oxford Dataset.
Our system is organized around missions, which in general will help us acquire more data due to gami cation aspects. Also, besides engaging users, missions provide additional data external to the image, that may be useful for aiding classi cation in the future.
Other future works include testing our algorithm in other species that contain large variations in the considered features (color, texture and shape), such as butter ies, shes, birds and so on, and evaluating more e cient classi ers.
We will also evaluate our system with real users analyzing how their behavior is a ected with gami cation and automatic classi cation techniques. Finally, we aim to use crowdsourcing for labeling training images, in order to build various datasets that should represent the local ora and fauna for diverse locations.