Snake species identi cation based on images is important to quickly treat patients su ering from snake bites using the correct antivenom. The SnakeCLEF 2020 challenge, which is part of the LifeCLEF research platform, is focused on this task and provides snake images and associated location information. This paper describes the participation of the FHDO Biomedical Computer Science Group (BCSG) in this challenge. The implemented machine learning work ow uses Mask Region-based Convolutional Neural Network (Mask R-CNN) for object detection, various image pre-processing steps, E cientNets for classi cation as well as di erent methods to fuse image and location information. The best model submitted before the challenge deadline achieved a macro-averaging F1-score of 0.404. After the expiration of this deadline, the results could be improved up to a macro-averaging F1-score of 0.594.
This paper explains the participation of University of Applied Sciences and Arts
Dortmund (FHDO) Biomedical Computer Science Group (BCSG) at the
Conference and Labs of the Evaluation Forum (CLEF) 20204 SnakeCLEF challenge5
for snake species identi cation [
The identi cation of snake species is important as there are approximately
between 81,410 and 137,880 victims of snakebites dying every year [29]. These
deaths result from inaccurate knowledge about the species and consequently
about the antivenom needed [
The high diversity of snake species [27] and their partially similar appearances
lead to confusion [
Therefore, the target of the SnakeCLEF challenge is the improved and robust
identi cation of snake species based on photographs [
In this article, the experiments and results of FHDO BCSG are presented. For this reason, Section 2 describes previous work in this eld of research. Afterwards, the general machine learning work ow is illustrated in Section 4, followed by a description of the achieved results in Section 5. Finally, the results are summarized in Section 6. 2
Automated identi cation of snake species using machine learning is rarely studied, resulting from small datasets of annotated images.
James et al. [
As manual extraction of features describing the appearance of a snake is
tedious, recent articles used automated feature extraction, for example, texture
features [
Texture features were used in Amir et al. [
Patel et al. [
Abdurrazaq et al. [
Abeysinghe et al. [
As already mentioned, Gokula Krishnan [
The training dataset used in the actual SnakeCLEF and AICrowd Snake Species Identi cation Challenge round 4 consists of 245,185 red-green-blue- (RGB-) color-space-images (models trained on the training dataset were referred to as T1) assigned to 783 di erent snake species. Additionally, a validation dataset is 7 https://www.tropicalherping.com/, [last accessed: 2020-07-17] 8 https://apps.who.int/bloodproducts/snakeantivenoms/database/, [last cessed: 2020-07-17] ac14000 12000 10000 y c n ue8000 q e fr e t luo6000 s b A 4000 2000 0 ilsa )891 its 2 r , − 1 isph (n= o n m a h t /su t/aa /su t/su i/nn /aem i/sne ii/r / / / /su lsau )81= iilitttrccsknanodoga− iitrsab−m liitrrcscahphoaphooh−mm iiiirrcscanabnuoh− iffsahophgeo−m ijiiysauheb−m lffrycaenodo− irrzscvceehaepoa−wm llijdaapaan− iltttscsaunupnuahohp−m lliiirscsysaohnopb− ilililrscssspohdophooepa− itrcsycspheohhnopha−m (n t o b available, which includes another 14,029 images (models trained on the training and validation dataset were referred to as T2). The class distribution of the snake species is highly unbalanced as can be seen in the bar plot of the absolute class frequencies depicted in Figure 1.
An analysis of the dataset with AntiDupl9 revealed 1,713 duplicate images in
the training set. Some of these duplicates are associated with di erent species
like \Image not found" images, that are the result of download problems. Other
duplicates are correctly associated with several species as they depict distinct
9 https://github.com/ermig1979/AntiDupl, [last accessed: 2020-07-15]
snakes. When the mean squared di erence between images in AntiDupl is relaxed
to 2 %, another 2,114 duplicates can be found. These are the result of di erent
jpeg compression rates for the same image, resize operations or deletion of
copyright information. Another problem that has been found are out-of-class images,
that have been injected by the organizers. These images contain no snakes but
for example, ice-hockey players, churches, other animals, persons, and mangas.
To identify them for exclusion from the training set, a standard ImageNet [
The images are associated with metadata that provides information about the continent and country of the place where the image has been taken. For some snake depictions, the information is not given and only \UNKNOWN" is provided in the metadata. This information could be used for better classi cation. It should be noted, that the number of snake species in the dataset does not match the natural occurrence of a snake in a location. For example, the most frequent species with German country information in the dataset is pantherophis guttatus, the corn snake which is not endemic in Germany but is the pet snake number one in Germany. Accordingly, the data set takes into account that pet snakes can also attack humans. 4
This section describes the work ow used to learn a discrimination between the
di erent snake species. The generalized work ow is depicted in Figure 2. The
work ow is modular and in the course of the challenge, it was examined how
di erent implementations of the individual modules a ect the classi cation
performance on the test dataset. In this section, the components are described more
precisely and di erent implementations of them are demonstrated. The work ow
has been implemented using the programming language Python 3.6.9 [28] and
was based on Keras 2.2.4-tf [
Image pre-processing included an optional object detection stage and a mandatory stage, where rectangular images were transferred to a square shape afterwards. Additionally, the images were augmented, optionally branded using locational information and fed into the deep learning training network. Finally, an optional multiplication of the prediction probabilities and the a priori probability distribution of the snake species occuring at the corresponding location has been implemented.
Input image
Object detection
Image preprocessing Augmentation Image branding Model training Metadata multiplication
The idea of using an object detection stage before executing an image classi
cation was inspired by the winning team [
In order to train the object detection model, the annotated snake images
available from the winning solution of round 2 of the AICrowd Snake Species
Identi cation Challenge [
As most of the deep learning classi cation models expect input images of square shape and prede ned dimensions, it has been important to transform the mostly rectangular images, or image parts detected by the object detection into square shape and adjust the image for the expected image dimension of the classi cation model. There are di erent possibilities for the extraction of quadratic from rectangular images. The methods used in this paper are described in this section and the implemented combinations of this methods are summarized in Table 1. The results of the experiments achieved using di erent image pre-processing methods are summarized in Table 4.
Resize The least complex possibility has been to rescale images without consideration of aspect ratio. This resulted in highly distorted images so the texture and the shape of the snake have been disturbed especially for images with 10 https://github.com/DiffPro-ML/Mask_RCNN, [last accessed: 2020-06-30] 11 https://github.com/matterport/MaskRCNN, [last accessed: 2020-06-30] 12 https://github.com/tensorflow/models/blob/master/research/object_ detection/g3doc/tf2_detection_zoo.md [last accessed: 2020-08-03] strongly di erent image dimensions. In this paper, two rescaling procedures, one considering and one retaining the aspect ratio were compared to each other. In the latter case, images had to be padded with further information to transfer them to a square shape.
Scaling Another problem which occurs during pre-processing is the problem of upscaling. Upscaling small images lead to poor image quality. It has been suspected that this could bring di culties in texture recognition. In this paper, approaches which did and did not use upscaling for image pre-processing were compared to each other. If upscaling has been avoided, approaches were needed to pad pixel information for the remaining image sections.
Fill boundaries As previously mentioned, there were some di erent cases,
where padding was required to get input images with preset image dimensions.
One strategy to solve this issue has been to pad the image by a monochrome
color. Koitka and Friedrich [
In combination with object detection, it has been possible to increase the ROI and thus pad the image using background information instead of monochrome color. In this case, the image section prede ned by the object detection workow has been expanded as long as a quadratic section is found or one of the dimensions of the original image were smaller than the expected dimension of the square. If this happened, the average color of the image has been used to pad protruding boundaries. It has been attempted to include background evenly on all sides to center the snake. Sometimes this was not possible, for example, if the snake was located in a corner of the original image. In this case, the ROI has been moved to include background information of the remaining directions, thus the snake has not been centered in the image.
No consideration of the aspect ratio Up-scaling No padding Consideration of the aspect ratio Up-scaling Monochrome padding Consideration of the aspect ratio No up-scaling Monochrome padding Consideration of the aspect ratio No up-scaling Background padding
Data augmentation has been used to expand the training images and avoid over tting. In each epoch of the training process, the images were randomly transformed. These transformations included random cropping of approximately 10 % of the image pixels per dimension, a rotation in the range of 40 , a widthshift, height-shift, random shearing, zooming each with a factor of 0.2, as well as the possibility of horizontal ipping. If pixel positions were generated during this procedure, for which no image information has been available, those were lled using the value of the nearest available image position. During the challenge, the work ow has been adapted to speed up the image classi cation procedure. In the later version of the work ow those pixels used black as a monochrome color. 4.4
E cientNets As also used by Gokula Krishnan [
Various models of the E cientNets family were used in this competition
from E cientNet-B0 up to E cientNet-B4 networks (B0 - B4 in Section 5).
The results of using di erent models of the E cientNets family can be found
in Table 6. The model weights were initialized by a model pre-trained using
noisy student [30]. The E cientNets were extended by a atten layer, a dense
layer with 1000 neurons and Swish [22] as an activation function and a dense
layer with 783 neurons, which corresponds to the number of snake species and
softmax activation were added to the previous architecture. The described model
was trained for a few epochs on the snake classi cation task to warm-up the
network. In this phase only the newly added layers and the batch normalization
layers have been trained. Afterwards all layers were trained for a larger number
of epochs (N10+50 denotes a warm-up phase including ten epochs and 50 epochs
are used to train the entire model). Di erent batch sizes were used as further
parameters to train the model (32 is encoded as BS32, 64 as BS64 etc., BS64/32
means that a batch size of 64 has been used during the warm-up phase and a
batch size of 32 has been used afterwards). The chosen batch size depended on
the image size (e.g., an image size of 128 128 is encoded as S128 in Section 5) the
classi cation model and the available graphics processing unit (GPU) memory.
The results of models using di erent image sizes can be found in Table 5. The
learning rate ( ) was likewise adjusted depending on the batch size (LR1 encodes
a learning rate of 10 4 during the warm-up phase and 10 5 during ne-tuning
and LR2 encodes a learning rate of 10 5 during the warm-up phase and 10 6
during ne-tuning in Section 5). All submissions described in this paper used the
Adam optimizer ( 1 = 0:9, 2 = 0:999, = 10 7) [
Since the dataset of the challenge had very unbalanced class frequencies, different class weight functions were used in order to implement an oversampling. Equation 1 describes a linear class weight function (W1 in Section 5) and Equation 2 describes a function where very low frequencies were less oversampled (W2 in Section 5). For both equations, F (c) denotes the frequency of class c. For comparability reasons, one model has been trained without class weights. w1(c) = max F (c)
F (c) w2(c) = 1
1 q max F (c) + 0:5
F (c)
Polyak Averaging Polyak averaging, based on the approach of Polyak [
Wpolyak(i) = exp i 2 (1) (2) (3) 4.5
Optionally, location information was added to some models by multiplying the prediction probabilities of the classi cation model by the a priori probability of the snake class for the speci ed location (M1 denotes the multiplication of the locational distribution). The a priori probabilities were estimated by the relative frequency distribution of the snake species at the location in the training and validation dataset. Usually, the country information was used in this step, only if this information was missing, the distribution of the continent has been used instead. For some images, both country and continent information were missing. In those cases, the frequency distribution of the entire dataset has been used. The softmax function was applied after this multiplication, to normalize the results.
Another variant has been implemented based on the previously described procedure. The sole exception has been that the raw prediction probabilities of images with missing country and continent information were not multiplied (abbreviated as M2 in Section 5). As a second variation of this method, all prediction probabilities were multiplied by a binary variant of the frequency distribution, which thus denotes if a snake was or was not present at a location (M3 in Section 5). The results achieved using the di erent metadata integration strategies are summarizes in Table 8.
During the experiments of the FHDO BCSG a few alternatives have been investigated. These methods were only tested in small experiments and are not described in this paper for reasons of clarity.
Image Branding As an alternative to the simple multiplication of the
location distributions, an approach has been implemented, which directly adds the
location information into the classi cation network. This has been done using a
binary image branding technique introduced in Pelka et al. [
bw = d 8 4 (4) The rst box starts directly at the left border of the image and after every box, space was left for 4 pixels.
The continent information has been added as binary boxes at the top border of the image, while the country information has been added at the bottom border of the image. Since a distinction has been made between seven continents as well as the \unknown"-class, every box at the top of the image represents a continent (abbreviated as M4). A similar approach to encode the country information would result in small boxes because 189 countries had to be distinguished. The used image branding approach is illustrated in Figure 3. In this case, a binary encoding of the country index has been chosen, so that eight boxes could be used to represent 28 = 256 di erent countries. Hereafter, the combined branding of continent and country information is abbreviated as M5. 5
In this section, the classi cation results for the test dataset of the challenge are described. Table 2 summarizes the most relevant successful submissions of the FHDO BCSG for the SnakeCLEF challenge.
This table is mainly used to give an overview about the submitted models. In order to get a better insight into the partial results and the e ects of the di erent methods used, partial aspects are considered in individual tables in the further course of this section. It was possible to submit models to the AICrowd Snake Species Identi cation Challenge after the submission deadline of SnakeCLEF expired. Therefore Table 2 presents a few models which achieve better Original image
Branding lookup tables
Branded image Continent
Africa Continent encoding: …
Index Encoding 0 10000000 … … South America 6 00000010
Unknown 7 00000001
Country Afghanistan … Brazil … Zimbabwe results than the best submission in the SnakeCLEF challenge. In order to avoid miscommunication, the submissions in Table 2 are listed in chronological order and the deadline of the challenge is highlighted.
The results of the di erent object detection datasets are summarized in Table 3. This table only presents the parameters, which are necessary for this comparison. It should be noted, that all the other parameters of the compared models are identical, as can be veri ed in Table 2. This type of presentation is also used in subsequent tables.
The comparison of submissions 68418 and 68450, as re ected in Table 3 shows, that the macro-averaging F1-score (abbreviated as F1 hereafter) increased by 0.010 when additional images were annotated, whereas log loss remains stable. Moreover, the number of images where no snakes were identi ed decreases from 141 to 123 in the joined training and validation dataset.
Additionally, submission 68678 is a model, which was trained using no object detection. This model is not completely comparable to any other models, but submission 68632 di ers only in the image pre-processing step. Comparing those two models, shows a slightly better performance of the model which used the object detection. As previously mentioned, this comparison is not entirely fair.
Table 4 summarizes the results of models trained based on di erent preprocessing methods. As can be seen, pre-processing in uenced the classi cation results achieved for the test dataset. Remarkable was the good performance of the submission 68506 which used image resizing without consideration of the aspect ratio. This model achieved the best F1 of 0.452. It has been expected, that this image pre-processing would achieve bad results as major distortions were possible, so in some cases humans were not able to recognize snakes in those images.
The second-best result was achieved for submission 67962. In this submission,
the ROIs detected during object detection have been expanded and thus were
padded using background information. The submission reached an F1 of 0.403
Abbreviations: O1: Object detection dataset from [
ID Pre-processing pipeline F1 Log loss 68506 No aspect ratio, up-scaling, without padding (I1) 0.452 6.648 67727 Aspect ratio, up-scaling, monochrome padding (I2) 0.389 6.650 68432 Aspect ratio, no up-scaling, monochrome padding (I3) 0.369 6.650 67962 Aspect ratio, no up-scaling, background padding (I4) 0.403 6.650 Table 5 summarizes the o cial classi cation results achieved using di erent image sizes as model input. The results of the comparison corresponds to other experiments executed during the challenge and shows that models trained on larger image input sizes achieved better classi cation results. Increasing the image size from 128 128 to 196 196 boosted the F1 by approximately 0.080. The used image sizes may look striking, because E cientNet-B0 models are usually trained using images including 224 224 pixels and E cientNet-B4 models are optimized for an image size of 380 380 pixels. The use of small images in this approach resulted from the fact that some early submissions failed because of memory issues. The problem has been xed after the deadline of the SnakeCLEF challenge expired. Some of the later submissions used larger image sizes consistent to the original E cientNets input sizes and thus achieved better results.
Next, the in uence of di erent model architectures on the classi cation results were investigated. In Table 6, a comparison is presented concerning di erent model architectures. The comparison shows, concurrently to some experiments not listed here for reasons of clarity, increased F1 for upscaled models. Submission 67727, which was based on an E cientNet-B2 architecture outperformed submission 67700 by an increase of the macro averaging F1-score of 0.037. Submission 68541, which represents an E cientNet-B4 architecture, achieved an F1 of 0.426 and thus outperformed the results of submissions 67727 and 67700 by 0.037 and 0.074. It should be noted that all of the submissions summarized in Table 6 were trained using an image size of 128 128 pixels which is due to some memory issues already mentioned before.
Some additional experiments were performed comparing di erent top layer architectures, for lack of time those were not completely comparable to each other, especially because the number of epochs used for training di ered for most of the models. For this reason these results are not elaborated in this paper.
ID Model architecture F1 Log loss 67700 E cientNet-B0 (B0) 0.352 6.651 67727 E cientNet-B2 (B2) 0.389 6.650 68541 E cientNet-B4 (B4) 0.426 6.648
It has been mentioned in Section 4 that di erent weight functions can be used to overcome unbalanced class distributions. The results of submissions 67727, 67882 and 67901 show, that the function introduced in Equation 2, which has been used in submission 67727 achieved a macro averaging F1-score of 0.389 and thus outperformed submission 67882, which used a linear class weight function and achieved an F1 of 0.365 and submission 67901, which used no class weights and achieved a macro averaging F1-score of 0.377. One possible reason for the poor results of the linear weighting could be the high di erences in class frequencies, which lead to larger weights for rare classes.
The results of the dataset ltering strategies, which are presented in Table 7, have been inconclusive. For the work ow used in submissions 67675 and 67700, the model trained on the reduced dataset performed worse than the model trained on the complete dataset. The opposite behaviour has been observed for the work ow used in submissions 67696 and 67727, which achieved F1 of 0.392 and 0.389. As the ltering removed images from the training dataset, where no snakes were present and no clear bene t is reached using this ltering, one could assume, that there might be some images in the test dataset where no snakes are present.
ID Strategy for dataset ltering F1 Log loss 67675 Duplicates and plausibility ltering (D1) 0.338 6.652 67700 No ltering (-) 0.352 6.651 67696 Duplicates and plausibility ltering (D1) 0.392 67727 No ltering (-) 0.389
As can be noted in Table 2, most of the earlier submissions, achieved high log losses of about 6.6, while others had log losses of about 1 with a confusing dependency to the achieved macro averaging F1-scores. This problem appeared because of softened prediction results if softmax normalization is used after the multiplication of the location frequencies and has been xed using maximumnormalization instead (e.g., submissions 68520, 68574, 68575 and 68655).
The results of adding metadata to improve image classi cation are described in Table 8. It can be seen, that adding metadata to a model by multiplying the model results using the a priori probability of the snake class for the given location lead to an increased F1. Submission 68574, which was trained using the same work ow as submission 68520, except adding metadata, outperforms this model by an increase of F1 of 0.109. As can be seen looking at submission 68575 those results can be further improved by a value of 0.016, if the multiplication is only used for available country and continent information. Submission 68655 exhibited a similar result of 0.445 using binary information about the availability of a species in a country or continent.
The results of the submissions 69365, 69750, 69768 and 69849 show that models which used the image branding presented in Section 4, achieved no bene t in comparison to multiplying the raw predictions by the location information. It can be noted, that submission 69750, which combined country and continent branding achieved a poor F1 of 0.361, whereas the model, which used only continent branding (submission 69768), achieved a better F1 of 0.437. One possible reason for this might be the use of the complex positional encoding of the country information which is hard to learn for a CNN, which focuses more on local di erences. Because of the limited time, it was not possible to investigate this problem more thoroughly, thus additional investigations should be under examination in future work. The combination of continent branding and multiplication of the a priori probability distribution in submission 69849 achieved similar results than the model, which used no branding, but the multiplication. 69365 Multiplication (M1) 0.460 69750 Branding continent and country (M5) 0.361 69768 Branding continent (M4) 0.437 69849 Branding continent and multiplication (M4+M1) 0.459
The best model submitted before the SnakeCLEF deadline expired was submission 68023 which was based on an E cientNet-B4 model architecture and achieved an F1 of 0.404. Due to the previously mentioned memory issues, this model used an unusual small image size of 128 128 pixels. The newly added layers of the described model were trained with a warm-up phase of ten epochs and another 50 epochs were used to ne-tune the entire model. During the training process a batch size of 64, a learning rate of 0.0001 and Adam optimizer have been used. Location information was added using the described multiplication procedure. Polyak averaging with exponential weights has been used to combine the results of the last ve training epochs. The Polyak averaging achieved an improvement of F1 of 0.001 in comparison to submission 67734 which used no Polyak averaging.
The best submission after the SnakeCLEF deadline expired was submission 69888 which achieved a macro-averaging F1-score of 0.594 and a log loss of 1.064. The main di erences in comparison to the best model before the deadline expired were, that the model was trained using the prede ned image dimensions of an E cientNet-B4, which are 380 380 pixels. Due to the increased image size, a smaller batch size of 13 and a decreased learning rate has been used. Furthermore, 109 instead of 50 training epochs have been applied, and the location distribution was multiplied only for known countries and continents. The model included no Polyak averaging. 6
In conclusion, it can be stated that snake species identi cation is a challenging task, primarily because of the high diversity of snake species, high intra-class variance, and low inter-class variance.
The main improvements in snake species classi cation presented in this paper are based on increasing image size, combining location and image information as well as upscaled model architecture. The results presented in this article show improved classi cation results using an object detection strategy previously to the image classi cation. However, a plausibility ltering of the training dataset showed no clear improvement. Some di erences were detected in dependence of the pre-processing steps. Nevertheless, no clear insights could be achieved about which steps are particularly promising for good classi cation results. The implementation and application of the di erent pre-processing steps turned out to be relatively time-consuming. Besides, it has been previously mentioned, that there were some memory issues which lead to a focus on small image sizes as well as less upscaled model architectures in the early course of the challenge. Thus the time needed to optimize the classi cation parameters more precisely and to try out di erent optimizers was reduced. It is expected, that the results may be further improved by adjusting those parameters.
Due to the use of AICrowd as a submission platform, it has been possible to test a large number of di erent models. This enables to get direct feedback about the performance on the test dataset, and thus gives a good estimate about which methods gets the most promising results. In addition it facilitates the comparison between teams before the deadline expires. In this article, it has been mentioned before, that there were some memory issues which were related to the architecture of the test environment. In some cases debugging has been complicated because the logs were not accessible. These concerns were compensated by the very prompt and useful help from the organizers.
The work of Louise Bloch and Obioma Pelka was partially funded by a PhD grant from University of Applied Sciences and Arts Dortmund, Germany.
The authors want to thank Raphael Brungel for the constructive proofreading of the manuscript. 22. Ramachandran, P., Zoph, B., Le, Q.V.: Searching for activation functions. Computing Research Repository (CoRR) abs/1710.05941 (2017), http://arxiv.org/ abs/1710.05941 23. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards realtime object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6), 1137{1149 (2017). https://doi.org/10.1109/tpami.2016.2577031 24. Ruppert, D.: E cient Estimations from a Slowly Convergent Robbins-Monro Process. Tech. rep., School of Operations Research and Industrial Engineering, Cornell University, Ithaca, NY (02 1988) 25. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 211{252 (2015). https://doi.org/10.1007/s11263-015-0816-y 26. Tan, M., Le, Q.: E cientNet: Rethinking Model Scaling for Convolutional Neural Networks. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. vol. 97, pp. 6105{6114. Long Beach, California, USA (06 2019), http://proceedings.mlr.press/v97/tan19a.html 27. Uetz, P., Hallermann, J., Hosek, J.: The Reptile Database 2019, http://reptiledatabase.reptarium.cz, [last accessed: 2020-06-10] 28. Van Rossum, G., Drake, F.L.: Python 3 Reference Manual. CreateSpace, Scotts
Valley, CA, 1 edn. (2009) 29. World Health Organization (WHO): Snakebite envenoming - Key Facts 2019 (2019), https://www.who.int/news-room/fact-sheets/detail/snakebiteenvenoming, [last accessed: 2020-06-10] 30. Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with Noisy Student improves ImageNet classi cation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10687{10698 (06 2020)