Recommender systems meet species distribution modelling Indre Zliobaite Department of Computer science, University of Helsinki, Finland Abstract Recommender systems techniques can naturally lend themselves to species distribution modelling if biological species are treated as items and places where they occur are treated as users. In this setting recommendation scores can reflect which habitats are suited for which species. Recommendation scores can also be used for reconstructing relative abundances of species, and analysing their rises and declines over millions of years in the past. Analysis of such predictions can shed light on the effects of changing environments on the biosphere now and in the past, as well as help to make predictions for the future. The major potential advantage of the recommender systems treatment over many existing solutions is the large spatial and temporal scale at which such analysis can be done within a single model. A single model makes predictions easier to compare globally in space and over time. While algorithmic applica- tion of recommender systems techniques to species distribution modelling is relatively straightforward, model selection and evaluation is particularly challenging, as there is no possibility for online tests or on-demand sampling, since the past worlds are long gone. Explainability is paramount in these tasks. Here we highlight the main challenges and promising directions of evaluation of such modelling, which is still in early stages of development. We show how aggregated prediction statistics and constraints may help for reliable model selection and evaluation. We illustrate the approaches on a case study of the mammalian fossil record from Europe around 8-17 millions of years ago. Keywords matrix factorization, implicit feedback, species distribution modeling, NOW database 1. Introduction Species distribution modelling is at the centre of ecology research and conservation applications [1, 2]. Typically, the computational task is to predict the probability of occurrence of biological species as a function of environmental conditions, as well as co-occurrence of species [3, 4]. Species distribution modelling is thriving for analysing the biodiversity on Earth today, and gaining popularity for analysing organismic evolution of the past [5, 6]. Most species distribution models in ecology focus on predicting the probabilities of occur- rence for one species at a time, even if species co-occurrences are taken into account as inputs. Such models are primarily used for understanding habitats of individual species. Biodiversity management and conservation applications, however, require an integrative view over ecosys- Perspectives on the Evaluation of Recommender Systems Workshop (PERSPECTIVES 2021), September 25th, 2021, co-located with the 15th ACM Conference on Recommender Systems, Amsterdam, The Netherlands " indre.zliobaite@helsinki.fi (I. Zliobaite) ~ https://www.zliobaite.com/ (I. Zliobaite)  0000-0003-2427-5407 (I. Zliobaite) Β© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) tems. While multispecies distribution models, modelling several species at a time, are coming about [7, 8], their primary focus is still on modeling ecological niches across environmental gradients. This relies on high resolution climatic data, which realistically is not available for the record of distant past. Collaborative filtering techniques of recommender systems can be used to model preferences of organisms for different habitats without explicitly characterizing those habitats. This approach would extract patterns of species co-occurrence and extrapolate them over large spatial and temporal scales. Modelling the dynamics of an ecosystem with a single model makes the predictions directly comparable across different species and different biodiversity spots. The inferred model then can be used: 1. for identifying species that would do well but are likely to be missing at sites; 2. for reconstructing relative population sizes from species lists (this is similar to predicting product ratings from transactional data); 3. for tracking co-occurrences and co-evolution over time and space, and for analyzing macroevolutionary processes. Technical research in recommender systems for species distribution modeling is in early stages, but has already shown promising results [9]. 2. Recommender systems for species distribution modelling The recommender system task naturally lends itself to analysis of ecosystems if we consider species as items and sites where they occur as users. The relationship is not the other way around, because each user can consume only a limited number of products, but each product can be consumed by potentially infinite number of users. Similarly, each site can accommodate a limited number of species, but each species potentially can occur in an infinite number of sites. While both species and sites can potentially be described by features, here we consider the simplest scenario where only occurence information is available. Thus, we are in the collaborative filtering task setting. Data for such analysis of ecosystems can come from many sources. Many biodiversity databases maintain records of species diversity today, the majority of them aggregate data from many sources1 [10]. Fossil databases, describing ecosystems of the past, are also widely available [11]. There are also databases that aggregate biodiversity databases, for instance, GBIT2 . Some data may come from expert surveys, research expeditions or professional wildlife monitoring projects, another portion of data may come from citizen observations. Naturally, even within a single database data can be of varying quality, which is not unlike user data typically used for recommender systems. Most notably, the uncertainty of absences is higher than the uncertainty of presences. If a species has been not been reported at a place it is uncertain whether it does not occur there, or it just has not been encountered yet. This applies to species occurrence data at present as well as in the fossil record. Similarly, presence of a transaction in typical data for 1 https://en.wikipedia.org/wiki/List_of_biodiversity_databases 2 https://www.gbif.org/ recommender systems signals that a user preferred a particular product, while absence of a transaction might mean either that the user did not like the product or has never came across it. Further challenges arise due to sparsity of data, especially that of the past ecosystems. Each fossil site presents only a tiny fraction of all species that have ever lived. Similarly, one user can realistically watch only a small fraction of movies that have ever been made. The total number of movies can vary from user to user as the number of species can vary from site to site. Last but not least, synonymy is a challenge for modelling user preferences as it is for analysing the fossil record. Just as the same movie may appear under different titles in different contexts (or countries), the same species can appear under different names in different research communities. Recommender system techniques are generally equipped to be robust to these shared challenges and hopefully can lend their perspectives to species distribution modelling. Sometimes information on abundances of species at sites might be available. This corresponds to availability of user ratings. Yet availability of relative abundances at large scales is rare. Typically, only lists of species that have occurred at sites are given. This corresponds to transactional user data without ratings. The latter setting calls for recommender systems solutions with implicit feedback [12, 13]. Such solutions may draw on the repetitiveness of a transaction or the certainty associated with a transaction in general. In the species modelling world certainty can be quantified via qualifiers associated with species identification. Fortunately, neither certainty nor presence-absence information has to be complete; recommender systems typically operate on incomplete information and such is the nature of information about species occurrences. 3. Evaluation challenges and evaluation criteria Latent factor models [12, 14, 15, 16] largely dominate the collaborative filtering research for over a decade due to their simplicity and effectiveness. For our case study consider a weighted latent factor model (WFM) [12] for collaborative filtering with implicit feedback. Let Dπ‘šΓ—π‘› be a binary matrix of observed presence or absence of taxa at sites. WFM defines a confidence matrix as C = 1 + 𝛼D, where 𝛼 is a parameter that accounts for asymmetry of uncertainties about presence and absence. The higher 𝛼 the more certainty is put on presences in contrast to absences. WFM factorizes the occurrence matrix into two preference matrices taking into account C confidences of the transactions D βˆ’ β†’ Xπ‘šΓ—π‘˜ Γ— Yπ‘›Γ—π‘˜ T . Here π‘˜ is a parameter specifying the dimensionality of the projection. WFM minimizes cost function minπ‘₯⋆ ,𝑦⋆ 𝑒,𝑖 𝑐𝑒𝑖 (𝑑𝑒𝑖 βˆ’ 𝑋𝑒T π‘Œπ‘– )2 + πœ† 2 𝑖 ||π‘Œπ‘– || . 2 βˆ‘οΈ€ (οΈ€βˆ‘οΈ€ βˆ‘οΈ€ )οΈ€ 𝑒 ||𝑋|| + Here 𝑐 and 𝑑 are elements of matrices C and D (defined earlier), and 𝑋 and π‘Œ are rows of matrices X and Y. πœ† a regularisation parameter. All in all WFM requires setting four parameters: 𝛼, π‘˜, πœ† and the number of iterations for minimizing the cost function. Next we need to define quantitative evaluation criteria and and a testing procedure for choosing the parameter values. While many indirect evaluation approaches exist for recommender systems [17], usually the most reliable is online testing, where users are exposed to different recommender solutions at random. Yet, since species occurrence data is almost always exclusively observational, online evaluation is not an option and we are left to evaluate the model fit based on the observational data used for modelling. If we wanted the model to reconstruct the observational data as closely as possible, the best approach would be to set the number of internal dimensions π‘˜ as high as possible and to set the regularisation parameter πœ† to zero. Such a model would memorise and reconstruct underlying data perfectly but it would not have predictive power, since it would overfit. Cross-validation, that would normally be used in predictive modeling to avoid overfitting, is not an option since there is no easy way to hold out a separate testing set. Variants of cross-validation have been used for testing autoencoder-based collaborative filtering [18, 19]. They would leave out some users for testing, which is possible with autoencoders, since they have explicit inputs to the model and outputs. That does not straightforwardly apply to latent factor models, however. For latent factor models we can do pseudo-cross-validation, where individual occurrences are nullified at random [16], and check which parameter settings best reproduce the nullified occurrences. Yet, this is not sufficient either. If we were simply to maximise this leave-one-out accuracy, the optimal solution would be to predict everything as ones, that is to predict all species to occur everywhere. Clearly, this is not an informative outcome either. Ideally, we want the model not only to reproduce observed occurrences but also to identify the species that are most likely to be missing at sites, as well as flag potential misidentifications. Thus, predictions must be inaccurate with respect to the training data in order to produce meaningful predictions. Our approach [9] thus is to push the model (1) to predict more occurrences than in the original data while at the same time (2) reproduce the occurrences in the original data reasonably accurately. The first criterion – pushing the model to predict more positives than in the original data is easy to achieve by increasing 𝛼. However, at the same time it is important not to overshoot the carrying capacity of the environments. Species-area relationships are restrictive in a sense that environments can accommodate only a limited number of species [20], which is somewhat predictable from the climatic conditions [21]. While a movie recommender system could potentially keep recommending highly scored movies to the user for as long as the user keeps watching them, an informative species distribution model should recommend a finite number of species that can exist on a site, and this number will certainly vary from site to site. An evaluation criteria that can be used to control for the model realism from this perspective could be requiring that the total-number of recommended species does not exceed, say, 20% of the species that are already there. The second criterion – keeping the occurrences in the original data accurately reconstructed should rely on a subset of data points for which we have high confidence of both positive occurrences and absences. Repetitive presences (if any of those are reported) can be considered as true positives. Absences out of the time range when the species is known to have been extinct (or has not originated yet) can be considered as true negatives. The latter requires a temporal information in the meta data and thus is primarily suitable for fossil data. With these two targets in mind one can aim at maximising a conventional evaluation metric, for example, the area under curve (ROC), on a subset of the data that only includes true positives and true negatives and let the aggregated statistics of positive predictions over sites and species take care of not deviating too far from the carrying capacity limits. 4. A case study Our case study shows an application of matrix factorisation with implicit feedback to recon- structing relative abundances of large plant eating mammals in Europe from about 17 to about 8 million years ago. This time range captures sites assigned to the European Land Mammal biozones from MN4 to MN12 [22], that include a major faunal turnover. Species occurrence data comes from a public fossil mammal database called NOW [23]. The database records sites where fossils have been found. Age information is assigned to sites, not to individual fossils. Each site has a list of species that have been found there. Some identifications of species may be uncertain, the database records uncertainty qualifiers. The database also records features that characterize each species, but we have not used this information in this study. We aggregated the data at the genus level rather than analyzing it at the species level. There is no difference from the algorithmic perspective, but this way the results are easier to interpret from the ecology perspective. Details of preprocesing can be found in [9]. The preprocessed dataset contains 104 genera (items), 351 sites (users) and 2616 occurrences (transactions). Sparsity is 93%. The preprocessed dataset used for this case study is available on GitHub3 . Following the principles outlined in the previous section we monitored the following quanti- tative performance measures: 1. 𝑃 Λ† all mean prediction score over all the data (the prior probability over all the data is π‘ƒπ‘Žπ‘™π‘™ = 0.063) [we want 𝑃 Λ† π‘Žπ‘™π‘™ to be slightly higher than 𝑃all , but not too much higher]; 2. MAE all mean absolute error over all the data [we want it to be small, but not zero]; 3. MAE animals mean absolute error over the numbers of occurrences for the animals [we want it to be small, but not zero]; 4. MAE sites mean absolute error over how many animals each site hosts [this relates to the carrying capacity, we want the error to be small, but not zero]; 5. AUC all area under ROC on all training data [we want it to be close to one]; 6. 𝑃 Λ† pos+ is mean prediction score over true positives) [ideally, we want 𝑃 Λ† π‘π‘œπ‘ + = 1]; 7. 𝑠𝑑(𝑃ˆ pos+ ) the standard deviation; Λ† neg+ is mean prediction score over true negatives) [ideally, we want 𝑃 8. 𝑃 Λ† 𝑛𝑒𝑔+ = 0]; 9. 𝑠𝑑(𝑃ˆ neg+ ) the standard deviation; 10. π΄π‘ˆ 𝐢+ area under ROC on selected true positives and true negatives [we want it to be close to one]; 11. 𝑠𝑑(AUC + ) the standard deviation. We tested around 200 parameter setting variants via a grid search in the 3-dimensional model parameter space (𝛼, π‘˜, πœ†). We kept the number of model fitting iterations fixed to 10. Instead of testing on all true positives and true negatives we randomly selected 10 of each and repeated 10 times for each model. This saved computational costs and sidestepped the challenge of class imbalance. We initialised the factor matrices by drawing random values from the normal distribution with zero mean and unit variance. It took a couple of minutes to fit one model using ad hoc implementation in R suite on a commodity laptop. 3 https://github.com/zliobaite/fossilrec Table 2 gives the scores. We can see that there is no clear winning variant, but we see some strong relationships. A higher number of internal dimensions gives more of positive predictions. The lower number of inner dimensions generally gives more tiled predictions. Higher regularisation (πœ†) generally, but not always gives fewer of positive predictions. Higher 𝛼 generally gives higher mean absolute error, especially on animals. This parameter controls the asymmetry of uncertainties about presence and absence, high values emphasise presence over absence. In practice, it seems, that absences in the fossil record are more certain, then, for instance, absences in movie recommendations, thus there is no need for as high values of this parameter as recommended in the original WFM publication [12]. Our earlier experimentation with various cuts of the data from NOW database revealed that the parameter values are somewhat sensitive to the overall size of the training matrix, but not too sensitive. As compared to commercial applications of recommender systems, ecological analyses will have relatively small datasets. Our dataset in this case study has 104 genera (animals) and 351 sites, and the density of ones in the matrix is nearly 7%. We conclude the case study by predicting relative abundances for one fossil site. We use a model with the following parameter settings: 𝛼 = 10, π‘˜ = 10, πœ† = 10, and 10 iterations for model fitting. In principle, we can use recommendation scores coming from the model for predicting relative abundances for all 351 sites, but we can only do quantitative evaluation on one site for which we have additional information on bone counts [24]. thus, we know in relative relatively how many of which animals were found there. Due to complex preservation processes [25] the bone counts do not necessarily reflect the relative numbers of animals that lived there many millions of years ago, but as the first approximation for evaluation these counts are fine. Generally, this information is rare in ecological data. Here we use this information only for evaluation purposes, not for model training. The remaining question is how to convert recommendation scores into relative abundance percentages. We mapped preference scores to relative abundances at site 𝑗 as: (𝑝𝑖𝑗 βˆ’ 0.5) πœ‹π‘–π‘— = βˆ‘οΈ€ , (1) 𝑑𝑖𝑗 >0 𝑝𝑖𝑗 where 𝑝𝑖𝑗 is the preference score for species 𝑖 to occur at site 𝑗, coming from the model; 𝑑𝑖𝑗 is the presence-absence matrix, where 𝑑𝑖𝑗 > 0 means that we only sum taxa that are reported to be present at site 𝑗. The subtraction of 0.5 from the probability score is an arbitrary cutoff implying that preference scores below 0.5 signal absence. In this study we only analyse the relative abundances of animals that are present, but in principle, the recommender systems approach would allow the estimation of potential relative abundances of animals that are absent as well. The challenge is how to keep the total number of recommended animals contained and in line with the carrying capacity of the environment, as discussed earlier. This is an open question for further research. Table 1 shows the results. We see that the order from the most abundant to the least abundant animals (genera) is not too far off, but the predictions for rare animals are quite too high. Table 1 Predicting relative population sizes at fossil site Somosaguas Norte, Spain (14-16 million years old), the numbers of fragments found come from [24] . Animal genus # fragments found % fragments found recommendation score % predicted Gomphotherium 786 51% 0.88 31% Anchitherium 484 31% 1.02 33% Prosantorhinus 92 6% 0.40 <1% Tethytragus 39 3% 0.65 10% Micromeryx 34 2% 0.75 16% Heteroprox 6 <1% 0.66 10% 5. Concluding remarks Recommender systems approaches open new perspectives for analysing ecosystems and species distribution modelling. Reliable evaluation of such approaches is an open challenge. Here we outline several evaluation criteria that are based on domain knowledge about ecosystems. Hope- fully similar solutions can potentially be useful in user modelling applications of recommender systems as well. Curious is to learn that the two settings are more similar than it may look from the first sight. Our case study showed that an off-the-shelf matrix factorisation approach already works reasonably well for fossil species distribution modelling, but many methodological challenges remain. Open directions for future research include taking time and energy constraints into the models. As not all species are alive at all times, models could take constraints of species being alive into their optimisation criteria. As ecosystems vary in energy (for example, tropical forests produce much more edible biomass than semi-deserts), models could incorporate such constraints as well. In the product world this would correspond to one customer having much more purchasing power than another. At a larger scale, different epochs with different climates may be considered as different contexts, where context-aware recommender systems [26] can offer better treatment. Finally, there is a large potential for blending occurrence information with descriptive features of animals and sites, drawing on recent works in reconstructing past environments [27, 28, 29, 30, 31]. The ultimate purpose is to understand how the living world was in the past, when is it livable, and how it works in general. 6. Acknowledgments The author is grateful to two anonymous reviewers for insightful feedback. Research leading to these results was partially supported by the Academy of Finland (grants no. 314803, 341623). References [1] J. Elith, J. Leathwick, Species distribution models: Ecological explanation and prediction across space and time, Annual Review of Ecology, Evolution, and Systematics 40 (2009) 677–697. [2] R. Pearson, Species? distribution modeling for conservation educators and practitioners, Lessons in Conservation 3 (2010) 54–89. [3] L. Pollock, R. Tingley, W. Morris, N. Golding, R. O’Hara, K. Parris, P. Vesk, M. McCarthy, Understanding co-occurrence by modelling species simultaneously with a joint species distribution model (JSDM), Methods in Ecology and Evolution 5 (2014) 397–406. [4] G. Tikhonov, A. Opedal, N. Abrego, A. Lehikoinen, M. d. Jonge, J. Oksanen, O. Ovaskainen, Joint species distribution modelling with the r-package Hmsc, Methods in Ecology and Evolution 11 (2020) 442–447. [5] C. Myers, A. Stigall, B. Lieberman, PaleoENM: applying ecological niche modeling to the fossil record, Paleobiology 41 (2015) 226–244. [6] S. Varela, J. Lobo, J. Hortal, Using species distribution models in paleobiogeography: A mat- ter of data, predictors and concepts, Palaeogeography, Palaeoclimatology, Palaeoecology 310 (2011) 451–463. [7] P. Dunstan, S. Foster, R. Darnell, Model based grouping of species across environmental gradients, Ecological Modelling 222 (2011) 955–963. [8] F. Hui, S. Taskinen, S. Pledger, S. Foster, D. Warton, Model-based approaches to uncon- strained ordination, Methods in Ecology and Evolution 6 (2015) 399–411. [9] I. Zliobaite, Recommender systems for fossil species distribution modelling, under review (2021). [10] J. Ball-Damerow, L. Brenskelle, N. Barve, P. Soltis, P. Sierwald, R. Bieler, R. LaFrance, A. Arino, R. Guralnick, Research applications of primary biodiversity databases in the digital age, PLoS ONE 14 (2019) e0215794. [11] M. Uhen, A. Barnosky, B. Bills, J. Blois, M. Carrano, M. Carrasco, et al, From card catalogs to computers: databases in vertebrate paleontology, Journal of Vertebrate Paleontology 33 (2013) 13–28. [12] Y. Hu, Y. Koren, C. Volinsky, Collaborative filtering for implicit feedback datasets, in: Proceedings of the 8th IEEE International Conference on Data Mining, IEEE ICDM, 2008, pp. 263–272. [13] K. Verstrepen, Collaborative Filtering with Binary, Positive-only Data, Ph.D. thesis, Uni- versiteit Antwerpen, 2015. [14] P. Gopalan, J. Hofman, D. Blei, Scalable recommendation with hierarchical poisson factor- ization, in: Proc. of the Thirty-First Conference on Uncertainty in Artificial Intelligence, 2015, pp. 326–335. [15] R. Salakhutdinov, A. Mnih, Probabilistic matrix factorization, in: Advances in neural information processing systems, 2008, pp. 1257–1264. [16] X. Ning, G. Karypis, Slim: Sparse linear methods for top-n recommender systems, in: IEEE International Conference on Data Mining, ICDM, 2011, pp. 497–506. [17] J. Herlocker, J. Konstan, L. Terveen, J. Riedl, Evaluating collaborative filtering recommender systems, ACM Trans. Inf. Syst. 22 (2004) 5–53. [18] D. Liang, R. Krishnan, M. Hoffman, T. Jebara, Variational autoencoders for collaborative filtering, in: Proc. of the 2018 World Wide Web Conference, WWW’18, 2018, pp. 689–698. [19] H. Steck, Embarrassingly shallow autoencoders for sparse data, in: Proc. of the 2019 World Wide Web Conference, WWW’19, 2019, pp. 3251–3257. [20] S. Cain, The species-area curve, The American Midland Naturalist 19 (1938) 573–581. [21] H. Hillebrand, On the generality of the latitudinal diversity gradient, The American Naturalist 163 (2004). [22] F. Hilgen, L. Lourens, J. van Dam, The neogene period, in: F. Gradstein, J. Ogg, M. Schmitz, G. Ogg (Eds.), The Geologic Time Scale 2012, Elsevier, 2012, pp. 923–978. [23] The NOW Community, New and old worlds database of fossil mammals (now), Licensed under CC BY 4.0, http://www.helsinki.fi/science/now/, 2020. [24] M. Domingo, D. Martin-Perea, L. Domingo, E. Cantero, J. Cantalapiedra, B. Garcia-Yelo, A. Gomez-Cano, G. Alcalde, O. Fesharaki, M. Hernandez-Fernandez, Taphonomy of mam- malian fossil bones from the debris-flow deposits of somosaguas-north (middle miocene, madrid basin, spain), Palaeogeography, Palaeoclimatology, Palaeoecology 465 (2017) 103–121. [25] A. K. Behrensmeyer, S. M. Kidwell, R. A. Gastaldo, Taphonomy and paleobiology, Paleobi- ology 26 (2000) 103–147. [26] G. Adomavicius, B. Mobasher, F. Ricci, A. Tuzhilin, Context-aware recommender systems, AI Magazine 32 (2011) 67–80. [27] L. Liu, K. Puolamaki, J. Eronen, M. Mirzaie Ataabadi, E. Hernesniemi, M. Fortelius, Dental functional traits of mammals resolve productivity in terrestrial ecosystems past and present, Proceedings of the Royal Society B 279 (2012) 2793–2799. [28] M. Fortelius, I. Zliobaite, F. Kaya, F. Bibi, R. Bobe, L. Leakey, et al., An ecometric analysis of the fossil mammal record of the turkana basin, Philosophical Transactions of the Royal Society: Biological Sciences 371 (2016). [29] A. Barr, Ecomorphology: Reconstructing cenozoic terrestrial environments and ecological communities, in: D. Croft, D. Su, S. Simpson (Eds.), Methods in Paleoecology: Reconstruct- ing Cenozoic Terrestrial Environments and Ecological Communities, Springer, 2018, pp. 339–349. [30] W. Vermillion, J. Head, P. Polly, J. Eronen, A. Lawing, Ecometrics: A trait-based approach to paleoclimate and paleoenvironmental reconstruction, in: D. Croft, D. Su, S. Simpson (Eds.), Methods in Paleoecology: Reconstructing Cenozoic Terrestrial Environments and Ecological Communities, Springer, Cham, 2018, pp. 373–394. [31] T. Faith, A. Du, J. Rowan, Addressing the effects of sampling on ecometric-based pale- oenvironmental reconstructions, Palaeogeography, Palaeoclimatology, Palaeoecology 528 (2019) 175–185. Table 2 Performance evaluation of models with different parameter settings. 10 best results within each evalu- ation criterion are highlighted in bold. 𝛼 π‘˜ πœ† ^ 𝑃 MAE MAE MAE AUC ^ pos+ 𝑃 𝑠𝑑 ^ neg+ 𝑃 𝑠𝑑 AUC + 𝑠𝑑 all all animals sites all ^ pos+ ) (𝑃 ^ neg+ ) (𝑃 (AUC + ) 5 5 5 0.05 0.235 9.74 5.194 0.9894 0.598 0.087 0.225 0.046 0.86 0.085 5 5 10 0.208 0.183 18.635 5.316 0.9826 0.561 0.082 0.075 0.032 0.964 0.054 5 5 15 0.131 0.133 8.51 2.709 0.9929 0.514 0.072 0.036 0.026 0.97 0.031 5 5 20 0.134 0.128 7.692 2.792 0.9914 0.443 0.047 0.025 0.021 0.966 0.032 5 10 5 0.296 0.258 42.048 12.333 0.9624 0.53 0.176 0.068 0.136 0.891 0.09 5 10 10 0.202 0.172 15.385 4.148 0.989 0.533 0.085 0.097 0.041 0.94 0.069 5 10 15 0.138 0.124 7.865 2.353 0.9957 0.512 0.045 0.024 0.029 0.988 0.016 5 10 20 0.14 0.128 8.221 2.749 0.9918 0.464 0.102 0.026 0.021 0.976 0.039 5 15 5 0.276 0.234 34.356 10.031 0.9734 0.51 0.146 0.161 0.088 0.854 0.135 5 15 10 0.155 0.131 8.346 2.011 0.9978 0.477 0.076 0.092 0.018 0.904 0.075 5 15 15 0.141 0.122 7.76 2.134 0.9966 0.463 0.069 0.035 0.023 0.96 0.053 5 15 20 0.133 0.123 7.721 2.726 0.9931 0.499 0.091 0.032 0.015 0.988 0.016 5 20 5 0.217 0.187 21.327 6.148 0.9853 0.43 0.085 0.108 0.102 0.85 0.136 5 20 10 0.183 0.149 9.952 2.464 0.9959 0.527 0.072 0.095 0.024 0.937 0.036 5 20 15 0.144 0.122 7.731 2.04 0.9969 0.468 0.071 0.035 0.018 0.97 0.028 5 20 20 0.137 0.126 8 2.684 0.9925 0.425 0.04 0.039 0.026 0.951 0.045 10 10 10 0.225 0.195 24.904 6.986 0.9954 0.581 0.068 0.105 0.055 0.909 0.055 10 10 15 0.173 0.156 17.308 4.439 0.9984 0.56 0.088 0.037 0.013 0.97 0.024 10 10 20 0.168 0.15 15.837 3.969 0.9977 0.602 0.105 0.028 0.027 0.984 0.019 10 10 5 0.093 0.234 13.115 5.749 0.9986 0.618 0.099 0.1 0.096 0.914 0.069 10 15 10 0.195 0.167 16.212 4.467 0.9985 0.544 0.063 0.074 0.055 0.949 0.058 10 15 15 0.172 0.146 13.577 3.47 0.9992 0.529 0.078 0.036 0.026 0.966 0.033 10 15 20 0.164 0.142 13.577 3.35 0.9987 0.54 0.048 0.03 0.03 0.984 0.025 10 15 5 0.216 0.211 28.154 8.353 0.988 0.544 0.09 0.073 0.145 0.893 0.083 10 20 10 0.174 0.145 10.712 2.952 0.9996 0.49 0.091 0.052 0.032 0.95 0.064 10 20 15 0.168 0.138 11.567 2.932 0.9995 0.549 0.068 0.034 0.029 0.962 0.039 10 20 20 0.164 0.138 12.76 3.068 0.9989 0.551 0.075 0.033 0.022 0.989 0.018 10 20 5 0.266 0.227 32.212 9.544 0.9826 0.527 0.062 0.109 0.108 0.88 0.072 10 5 10 0.196 0.19 27.519 7.407 0.9961 0.631 0.086 0.042 0.035 0.962 0.048 10 5 15 0.204 0.188 25.308 6.627 0.9936 0.602 0.077 0.04 0.04 0.982 0.025 10 5 20 0.187 0.176 20.856 5.444 0.9937 0.662 0.066 0.02 0.031 0.988 0.012 10 5 5 0.188 0.206 30.423 8.444 0.9959 0.665 0.093 0.085 0.077 0.961 0.036 20 5 5 0.238 0.254 49.625 14.51 0.997 0.737 0.05 0.007 0.064 0.983 0.03 20 5 10 0.236 0.242 47.375 13.798 0.997 0.744 0.072 0.03 0.045 0.992 0.012 20 5 15 0.237 0.234 45.279 13.063 0.9971 0.721 0.061 0.084 0.063 0.953 0.039 20 5 20 0.235 0.224 42.904 12.251 0.9965 0.721 0.075 0.045 0.036 0.98 0.034 20 10 5 0.185 0.209 29.856 8.823 0.9992 0.676 0.074 0.015 0.063 0.976 0.029 20 10 10 0.196 0.2 30.673 9.037 0.9995 0.669 0.071 0.044 0.051 0.942 0.053 20 10 15 0.196 0.193 29.058 8.484 0.9994 0.643 0.081 0.032 0.048 0.955 0.057 20 10 20 0.2 0.186 28.471 8.225 0.9993 0.678 0.061 0.036 0.054 0.989 0.02 20 15 5 0.16 0.181 19.423 5.761 0.9998 0.53 0.099 0.02 0.059 0.942 0.064 20 15 10 0.197 0.181 22.154 6.547 0.9997 0.541 0.097 0.014 0.046 0.952 0.044 20 15 15 0.178 0.167 20.106 5.895 0.9997 0.57 0.071 0.038 0.035 0.956 0.034 20 15 20 0.183 0.166 20.904 6.023 0.9997 0.613 0.108 0.033 0.032 0.964 0.039 20 20 5 0.198 0.182 18.625 5.519 0.9994 0.504 0.07 0.029 0.06 0.936 0.054 20 20 10 0.196 0.17 16.442 4.86 0.9996 0.569 0.087 0.083 0.048 0.921 0.098 20 20 15 0.169 0.15 14.894 4.396 0.9999 0.498 0.09 0.019 0.047 0.928 0.055 20 20 20 0.176 0.152 16.885 4.889 0.9999 0.584 0.099 0.043 0.027 0.976 0.035 30 5 5 0.264 0.283 62.096 18.376 0.9976 0.743 0.06 0.053 0.044 0.965 0.033 30 5 10 0.262 0.274 59.952 17.718 0.9973 0.774 0.1 0.021 0.079 0.982 0.036 30 5 15 0.265 0.267 57.769 17.031 0.9975 0.749 0.094 0.082 0.061 0.979 0.031 30 5 20 0.264 0.261 58.471 17.217 0.9975 0.746 0.104 0.084 0.06 0.969 0.038 30 10 5 0.2 0.226 36.481 10.809 0.9992 0.667 0.139 0.008 0.037 0.942 0.076 30 10 10 0.204 0.22 35.904 10.638 0.9995 0.66 0.078 0.055 0.064 0.934 0.057 30 10 15 0.213 0.215 37.038 10.957 0.9996 0.689 0.053 0.04 0.051 0.964 0.039 30 10 20 0.218 0.21 36.76 10.846 0.9994 0.67 0.104 0.037 0.059 0.95 0.089 30 15 5 0.166 0.194 21.683 6.442 0.9997 0.592 0.05 -0.006 0.043 0.96 0.053 30 15 10 0.176 0.187 23.375 6.926 0.9999 0.57 0.115 0.023 0.058 0.934 0.069 30 15 15 0.186 0.184 24.625 7.291 0.9999 0.621 0.093 0.029 0.038 0.953 0.064 30 15 20 0.196 0.182 25.75 7.607 0.9999 0.6 0.106 0.043 0.046 0.923 0.073 30 20 5 0.14 0.169 13.269 3.932 1 0.497 0.095 0.036 0.043 0.882 0.07 30 20 10 0.171 0.166 15.971 4.732 1 0.517 0.091 0.005 0.038 0.938 0.073 30 20 15 0.173 0.163 17.769 5.265 1 0.529 0.106 0.028 0.05 0.941 0.066 30 20 20 0.182 0.162 19.846 5.869 0.9999 0.579 0.068 0.027 0.045 0.976 0.029