Overview of GeoLifeCLEF 2018: location-based species recommendation

Abstract

The GeoLifeCLEF challenge provides a testbed for the systemoriented evaluation of a geographic species recommendation service. The aim is to investigate location-based recommendation approaches in the context of large scale spatialized environmental data. This paper presents an overview of the resources and assessments of the GeoLifeCLEF task 2018, summarizes the approaches employed by the participating groups, and provides an analysis of the main evaluation results.

Keywords

• LifeCLEF
• biodiversity
• big data
• environmental data
• visual data
• species recommendation
• evaluation
• benchmark

-

Automatically predicting the list of species that are the most likely to be observed at a given location is useful for many scenarios in biodiversity informatics. First of all, it could improve species identi cation processes and tools by reducing the list of candidate species that are observable at a given location (be they automated, semi-automated or based on classical eld guides or ora). More generally, it could facilitate biodiversity inventories through the development of location-based recommendation services (typically on mobile phones) as well as the involvement of non-expert nature observers. Last but not least, it might serve educational purposes thanks to biodiversity discovery applications providing functionalities such as contextualized educational pathways.

The aim of the challenge is to predict the list of species that are the most likely to be observed at a given location. Therefore, we provided a large training set of species occurrences, each occurrence being associated to a multi-channel image characterizing the local environment. Indeed, it is usually not possible to learn a species distribution model directly from spatial positions because of the limited number of occurrences and the sampling bias. What is usually done in ecology is to predict the distribution on the basis of a representation in the environmental space, typically a feature vector composed of climatic variables (average temperature at that location, precipitation, etc.) and other variables such as soil type, land cover, distance to water, etc. The originality of GeoLifeCLEF is to generalize such niche modeling approach to the use of an image-based environmental representation space. Instead of learning a model from environmental feature vectors, participants may learn a model from k-dimensional image patches, each patch representing the value of an environmental variable in the neighborhood of the occurrence (see Figure 1 below for an illustration). From a machine learning point of view, the challenge will thus be treatable as a multi-channel image classi cation task. The participants were provided with a train and test set of species geolocated occurrences. Both were rst composed of a .csv le with the occurrences spatial coordinates, the punctual values of environmental variables at the occurrence location, and, for the train table, the species name and identi er. Secondly, each row of the table (train and test) referred to a 33-channel image containing the environmental tensor extracted at that location. 2.1

Species occurrences

Occurrences data were extracted from the Global Biodiversity Information Facility platform (GBIF 6). To achieve precise species prediction from a geolocation, the geolocations in question must be as precise as possible. However, a high number of occurrences from the GBIF have a spatially degraded geolocation for conservation reasons. Thus, we have chosen source datasets with undegraded geolocations in France, which are : 1. Carnet en ligne from Tela Botanica. 2. Cartographie des Leguminosae (Fabaceae) en France from Tela Botanica. 3. Naturgucker dataset. 4. iNaturalist Research-grade Observations.

Only observations falling in the metropolitan French territory were kept so as to focus on a region for which we had an easy access to rich and homogeneous environmental descriptors for the whole dataset. Occurrences with uncertain names, as noti ed by the GBIF, were removed. The full dataset is nally composed of 291,392 occurrences. The labels to be predicted within the challenge are the species identi er ( eld species glc id). There are 3,336 species identi ers in total, and their associated taxonomic names are provided by the eld espece retenue bdtfx (bdtfx referential 4.1). Due to some unreferenced heterogeneity in the data collection protocol (naturalists checklists, conversion of site name to geolocation, etc), some geographical points accumulate several occurrences. Indeed, there are in total 75,668 distinct geolocations (with a maximum of 527 points in one geolocation). All occurrences geolocations are represented in Figure 2. It reveals the bias in the spatial distribution of the occurrences. 2.2

Environmental data

Each occurrence is characterized by 33 local environmental images of 64x64 pixels. These environmental images were constructed from various open datasets and include 19 bioclimatic quantitative variables at 1km resolution from Chelsea Climate [ 6 ], 10 pedological ordinal variables at 1km resolution from ESDB soil pedology data [ 11,12,16 ], one land cover categorical descriptor at 100 meters resolution from Corine Land Cover 2012 soil occupation data (version 18.5.1, 12/2016), one potential evapo-transpiration quantitative variable at 1km resolution from CGIAR-CSI evapotranspiration data ([18,19]), one elevation quantitative variable at 90 meters resolution from USGS Elevation data (Data available from the U.S. Geological Survey and downloadable on the Earthexplorer7) and one indicator of fresh water proximity at 12,5m resolution from the BD Carthage hydrologic data. As each of those variables are stored in large raster covering the French geographical territory. For any occurrence, we crop a 64 64 pixels window centered on the occurrence geolocation from the raster of each environmental variable. This way, we make the 64 64 33 environmental tensor 6 https://www.gbif.org/ 7 (https://earthexplorer.usgs.gov/) associated with this occurrence. Besides, the punctual environmental values associated with an occurrence, are simply the extracted cell's values from the rasters at the occurrence geolocation. The total of 291,392 occurrences were randomly split into a training set (218,543) and a test set (72,849) with the constraints that : { For each species in the test set, there is at least one observation of it in the training set. { An observation of a species in the test set is distant of more than 100 meters from all observations of this species in the train set to avoid major reporting dependencies.

Thus, the nal train set contained all of the 3,336 species, while the test set contained 3,209 species. 3

Task Description

For every occurrence of the test set, participants must supply a list of 100 species maximum, ranked without ex-aequo. The used evaluation metric is the Mean Reciprocal Rank (MRR). The MRR is a statistic measure for evaluating any process that produces a list of possible responses to a sample of queries ordered by probability of correctness. The reciprocal rank of a query response is the multiplicative inverse of the rank of the correct answer. The MRR is the average of the reciprocal ranks for the whole test set:

Q M RR = 1 X

Q

1 q=1 rankq where Q is the total number of query occurrences xq in the test set and rankq is the rank of the correct species y(xq) in the ranked list of species predicted by the evaluated method for the occurrence xq. 4

Participants and methods

22 research groups registered to the GeoLifeCLEF challenge 2018. Among this large raw audience, 3 research groups nally succeeded in submitting run les. Details of the used methods and evaluated systems are synthesized below and further developed in the working notes of the participants ([ 2 ], [ 15 ] and [ 8 ]). Table 1 reports the results achieved by each run as well as a brief synthesis on the methods used in each of them. Complementary, the following paragraphs give a few more details about the methods and the overall strategy employed by each participant.

FLO team, France, 10 runs, [ 2 ]: FLO developed four prediction models, (i) one convolutional neural network trained on environmental tensors (FLO 3). The CNN implemented a customized architecture. It also treated the categorical land cover descriptor independantly from quantitative variables for the primary layers. Activation's of both variables types where then fused in deeper layers. (ii) one neural network (FLO 2) trained on species occurrences falling at the closest spatial point and two other models only based on the spatial occurrences of species: (iii) a closest-location classi er (FLO 1) and (iv) a random forest tted on the spatial coordinates (FLO 4). Other runs correspond to late fusions of that base models, either by simply averaging either the output probabilities (FLO 5,FLO 6,FLO 7,FLO 8), or ranks with the Borda method (FLO 9,FLO 10).

ST team, Germany, 16 runs, [ 14 ]: ST experimented two main types of models, convolutional neural networks on environmental tensors with di erent data augmentations like rotation and ip of images (ST 1, ST 3, ST 11, ST 14, ST 15, ST 18, ST 19) and Boosted Trees (XGBoost) on vectors of environmental variables concatenated with spatial positions (ST 6, ST 9, ST 10, ST 12, ST 13, ST 16, ST 17). They also proposed a nearest-neighbor classi er based on the environmental variables of occurrences (ST 5), and two species cluster models (ST 17,ST 8) where groups of species are constituted by the similarity of the environmental variables where they occur. For analysis purposes, ST 2 corresponds to a random predictor and ST 7 to a constant predictor returning always the 100 most frequent species (ranked by decreasing value of their frequency in the training set).

SSN, India, 4 runs, [ 10 ]: SSN attempted to learn a CNN-LSTM hybrid model, based on a ResNext architecture [ 17 ] extended with an LSTM layer [ 3 ] aimed at predicting the plant categories at 5 di erent levels of the taxonomy (class, then order, then family, then genus and nally species). The four runs are derived from this model. 5

Results

We report in Figure 3 and Table 1 the main results achieved by the 33 submitted runs as well as some synthetic information about the used methods and variables for each run. The main conclusions we can draw from that results are the following: Convolutional Neural Networks outperformed boosted trees: Boosted trees are known to provide state-of-the-art performance for environmental modelling. They are actually used in a wide variety of ecological studies [ 4,1,7,9 ]. Our evaluation, however, demonstrate that they can be consistently outperformed by convolutional neural networks trained on environmental data tensors. The best submitted run that does not result from a fusion of di erent models (FLO 3), is actually a convolutional neural network trained on the environmental patches. It achieved a M RR of 0:043 whereas the best boosted tree (ST 16) achieved a M RR of 0:035. As another evidence of the better performance of the CNN model, the six best runs of the challenge result from the combination of it with the other models of the Floris'Tic team. Now, it is important to notice that the CNN models trained by the ST team (ST 1, ST 3, ST 11, ST 14, ST 15, ST 18, ST 19) and SSN team did not obtain good performance at all (often worse than the constant predictor based on the class prior distribution), which could be due to a mismatch of species identi ers, as noticed by the participant. For team ST, results can't be interpreted directly as a failure of the methods. The ranking of runs in the test set was not consistent with validation results and the learning process can be improved according to [ 15 ]. This illustrates the di culty of designing and tting deep neural networks on new problems without former references in the literature. Lastly, the approaches trying to adapt existing complex CNN architectures that are popular in the image domain (such as VGG [ 13 ], DenseNet [ 5 ], ResNEXT [ 17 ] and LSTM [ 3 ]) were not successfull. High di erence of performances in CNN learned with homemade architectures (F LO 6; F LO 3; F LO 8; F LO 5; F LO 9; F LO 10 compared to ST 3; ST 1) could underline the importance of architecture choices. Purely spatial models are not so bad: the random forest model of the FLO team, tted on spatial coordinates solely (FLO 4), achieved a fair M RR of 0:0329, close to the performance of the boosted trees of the ST team (that were trained on environmental & spatial data). Purely spatial models are usually not used for species distribution modelling because of the heterogeneity of the observations density across di erent regions. Indeed, the spatial distribution of the observed specimens is often more correlated with the geographic preferences of the observers than with the abundance of the observed species. However the goal of GeoLifeClef is to predict the most likely species to observe given the real presence of a plant. Thus, the heterogeneity of the sampling e ort should induce less bias than in ecological studies.

It is likely that the Convolutional Neural Network already captured the spatial information: The best run of the whole challenge (FLO 6) results from the combination of the best environmental model (CNN FLO 3) and the best spatial model (Random forest FLO 4). However, it is noticeable that the improvement of the fused run compared to the CNN alone is extremely tight (+ 0:0005), and actually not statistically signi cant. In other words, it seems that the information learned by the spatial model was already captured by the CNN. Besides, CNN uses the whole environmental tensor as input and is better than the XGBoost methods which used only the average of each environmental matrix as input. So it is likely that CNN captured more information than the average of the environmental image. It might be some patterns associated with a particular area, or more generic environmental patterns (a wet valley, etc.). The learning of species communities patterns has potential: We rst state that species have marked spatial patterns. Indeed, predicting the nearest species in space (FLO 1) or in the environmental space (ST 5) is much more e cient than simply listing species per global abundance (ST 7), which corresponds to a uniform prior on spatial distribution of each species. Second, methods that allow interactions between species abundance, either by building and predicting group of species that have similar environmental preferences (ST 17), or learning the association between species that co-occur in a close surrounding (FLO 2) perform better than simple nearest-neighbor approaches. However, these approaches are still limitating as, for example, FLO 2 only used the closest point as input information about surrounding species. Besides, even though the good performance of ST 17, there was very few groups of more than 1 species in their algorithm, which leaves small chances to predict non-common species while they represent the majority of species.

A signi cant margin of progress but still very promising results: even if the best MRR scores appear to be very low at a rst glance, it is important to relativize them with regard to the nature of the task. Many species (tens to hundred) are actually living at the same location so that achieving very high MRR scores is not possible. The MRR score is useful to compare the methods between each others but it should not be interpreted as for a classical information retrieval task. In the test set itself, several species are often observed at exactly the same location. So that there is a max bound on the achievable MRR equal to 0:56. The best run (FLO 3) is still far from this max bound (MRR=0:043) but it is much better than the random or the prior distribution based MRR. Concretely, it retrieves the right species in the top-10 results in 25% of the cases, or in the top-100 in 49% of the cases (over 3; 336 species in the training set), which means that it is not so bad at predicting the set of species that might be observed at that location. 8 ST 13 9 ST 10 10 ST 9 11 ST 12 12 ST 6 15 ST 17 16 FLO 2 13 FLO 4 14 FLO 7 19 FLO 1 17 ST 5 18 ST 8 20 ST 3 21 ST 1 22 ST 14 23 ST 7 24 ST 15 25 ST 19 0.4214 0.854 0.97 Spatial heterogeneity of model performances: We computed the MRR restricted to occurrences that fall in spatial quadrats of 10 10 km all over the French territory. We projected this on a map in Figure 4. The global performances of the methods hide spatial heterogeneity, as shown in the map. Indeed, Paris is the best predicted area, then the Mediterranean region and the Alpes. Then other regions like the Loire, the Pyrenees and the Atlantic coastline. One could think this is due to the larger number of points available in these areas, but this is not exactly true. Complementary analysis showed that the importantly sampled areas had a more stable MRR but not higher in average. Thus, improving models predictions should pass by nding reasons of varying regional performances, in the hope to bring a solution.

Rare species are not unpredictable: For each species and method, we calculated the MRR over the occurrences of this species in the test set. We ordered species per decreasing global occurrences count in the test set in order to compare the performances of each method along the gradient from common to rare species. The raw graphs were di cult to analyse because the MRR varies Fig. 4. MRR per 10 10km square spatial quadrat for FLO 3 over the study region. importantly for rare species, as there are very few occurrences. Thus, we operate a smoothing along the scarcity gradient. For each species we took the median of the MRR over the 40 species of closest rank on this scarcity gradient. Figures 5 and 6 show the result for FLO 3 (environmental CNN), ST 16 (XGBoost), FLO 4 (spatial Random Forest) and ST 7 (Global frequency of species). One can see that ST 7 early cancels along the scarcity gradient. This is because more than 50% of the species over which the median is calculated have a null MRR, which correctly represents the tendancy we want to observe. First, it seems that non-common species have marked spatial preferences because FLO 4 is much better when getting scarcer than ST 7. Second, the progression of predictions of FLO 3 and ST 16 compared to FLO 4 for rare species (in the long tail) suggests that those species mainly have marked environmental preferences that is not easy to capture with a spatial model which doesn't have access to this information. The CNN is very good at predicting non-common species, which may be a bit surprising as (i) its predictions should be smooth in space according to the width of some environmental images (64x64km for climatic and pedological variables) and the chosen architecture and (ii) rare species often have a restricted niche. 7

Conclusion

We have analyzed the results of the 3 participants of GeoLifeCLEF 2018. CNN models learnt on environmental tensors revealed to be the most performing method, however challenging to operate. According to those results, they are more e cient than Boosted Trees a state of the art method in species distribution modeling. This might be because they may detect particular area or environmental patterns as they access to the full surrounding environment data, but that remain to be proved. Spatial and species association methods have shown reasonably good results, but there is room for improvement, especially for the use of interdependence. The complementary analysis revealed that all methods had the same areas of unreliability. Furthermore, the integration of environmental variables seems to be very bene cial to the prediction of noncommon species. The task of nding the species found at a precise location is di cult because many species co-exist at very small spatial scales (under the meter). The accuracy of current geolocation devices doesn't even allow to indicate with this precision the point where the specimen was observed. Thus, in the future, the evaluation process shouldn't penalize predictions of other species that have been observed in such a close surrounding regarding the precision of the reported geolocation. 18. Zomer, R.J., Bossio, D.A., Trabucco, A., Yuanjie, L., Gupta, D.C., Singh, V.P.: Trees and water: smallholder agroforestry on irrigated lands in Northern India, vol. 122. IWMI (2007) 19. Zomer, R.J., Trabucco, A., Bossio, D.A., Verchot, L.V.: Climate change mitigation: A spatial analysis of global land suitability for clean development mechanism afforestation and reforestation. Agriculture, ecosystems & environment 126(1), 67{80 (2008)

References

1. De'Ath , G. : Boosted trees for ecological modeling and prediction . Ecology 88 ( 1 ), 243 { 251 ( 2007 )

2. Deneu , B. , Servajean , M. , Botella , C. , Joly , A. : Location-based species recommendation using co-occurrences and environment - geolifeclef 2018 challenge . In: CLEF working notes 2018 ( 2018 )

3. Gers , F.A. , Schmidhuber , J. , Cummins , F. : Learning to forget: Continual prediction with lstm ( 1999 )

4. Guisan , A. , Thuiller , W. , Zimmermann , N.E. : Habitat Suitability and Distribution Models: With Applications in R. Cambridge University Press ( 2017 )

5. Huang , G. , Liu , Z. , Weinberger , K.Q. , van der Maaten , L.: Densely connected convolutional networks . In: Proceedings of the IEEE conference on computer vision and pattern recognition . vol. 1 , p. 3 ( 2017 )

6. Karger , D.N. , Conrad , O. , Bohner, J., Kawohl , T. , Kreft , H. , Soria-Auza , R.W. , Zimmermann , N. , Linder , H.P. , Kessler , M. : Climatologies at high resolution for the earth's land surface areas . arXiv preprint arXiv:1607.00217 ( 2016 )

7. Messina , J.P. , Kraemer , M.U. , Brady , O.J. , Pigott , D.M. , Shearer , F.M. , Weiss , D.J. , Golding , N. , Ruktanonchai , C.W. , Gething , P.W. , Cohn , E. , et al.: Mapping global environmental suitability for zika virus . Elife 5 ( 2016 )

8. Moudhgalya , N.B. , Sundar , S. , Divi , S. , Mirunalini , P. , Aravindan Bose , C. : Hierarchically embedded taxonomy with clnn to predict species based on spatial features . In: CLEF working notes 2018 ( 2018 )

9. Moyes , C.L. , Shearer , F.M. , Huang , Z. , Wiebe , A. , Gibson , H.S. , Nijman , V. , MohdAzlan , J., Brodie , J.F. , Malaivijitnond , S. , Linkie , M. , et al.: Predicting the geographical distributions of the macaque hosts and mosquito vectors of plasmodium knowlesi malaria in forested and non-forested areas . Parasites & vectors 9(1) , 242 ( 2016 )

10. Nithish B Moudhgalya , Sharan Sundar , S.D.M.P. , Bose , C.A. : Hierarchically embedded taxonomy with clnn to predict species based on spatial features . In: CLEF working notes 2018 ( 2018 )

11. Panagos , P.: The european soil database . GEO: connexion 5(7) , 32 { 33 ( 2006 )

12. Panagos , P. , Van Liedekerke , M. , Jones , A. , Montanarella , L. : European soil data centre: Response to european policy support and public data requirements . Land Use Policy 29 ( 2 ), 329 { 338 ( 2012 )

13. Simonyan , K. , Zisserman , A. : Very deep convolutional networks for large-scale image recognition . CoRR abs/1409 .1556 ( 2014 )

14. Stefan Taubert , Max Mauermann, S.K.D.K. , Eibl , M. : Species prediction based on environmental variables using machine learning techniques . In: CLEF working notes 2018 ( 2018 )

15. Taubert , S. , Mauermann , M. , Kahl , S. , Kowerko , D. , Eibl , M. : Species prediction based on environmental variables using machine learning techniques . In: CLEF working notes 2018 ( 2018 )

16. Van Liedekerke , M. , Jones , A. , Panagos , P.: Esdbv2 raster library-a set of rasters derived from the european soil database distribution v2. 0. European Commission and the European Soil Bureau Network, CDROM , EUR 19945 ( 2006 )

17. Xie , S. , Girshick , R. , Dollar , P. , Tu , Z. , He , K. : Aggregated residual transformations for deep neural networks . In: Computer Vision and Pattern Recognition (CVPR) , 2017 IEEE Conference on . pp. 5987 { 5995 . IEEE ( 2017 )