Local Quality Improvement of Multispectral Imagery Classification with Radiometric‐spatial Feedback Iryna Piestovaa, Anna Kozlovaa, Artem Andreieva and Jan Rabcanb a Scientific Centre for Aerospace Research of the Earth, Institute of Geological Sciences, National Academy of Sciences of Ukraine, O. Gonchar str., 55-b, Kyiv, 01054, Ukraine b University of Zilina, Univerzitna, 8215/1, Zilina, 01026, Slovakia Abstract The essential requirement for accurate classification is the high resolution of input images. Among known classification problems, which caused by low-resolution images, are the mixing of training samples and the absence of boundaries between objects of different classes. The mentioned above problems were reduced by imagery spatial resolution enhancement and a hybrid approach to classification, which allows unmixing training samples and improving the quality of images and their classifications. Keywords 1 Remote sensing, spatial resolution enhancement, supervised classification, training samples clustering, multi-valued logic, subpixel reallocating 1. Introduction Classification is the process of dividing a mass of data into classes (groups) according to some criterion. In the process of computer classification, each pixel of the image is assigned to one of the selected classes. Classification of satellite images is a widely-used remote sensing tool for solving such tasks as land-cover mapping and change detection [1], forecasting gas and oil potential of subsoil plots [2], etc. For example, when compiling a map of mountain vegetation using the satellite image, you can divide the entire territory depicted on it into areas covered with forests, meadows, glaciers, etc. The resulting image is called a "classification map", or simply "classification". Computer classification is used to automatically separate objects displayed on images and obtain a map of the area. For computer processing, each image is presented as a table, each cell of which - an element of the image resolution (pixel) - contains a number indicating the brightness of this element. In multispectral imagery, the values of the brightness of objects are recorded in separate rather narrow areas - spectral bands - of the visible and infrared parts of the spectrum. Then, during computer processing, for each pixel of the image, several digital values of brightness are used at once in different spectral bands. The main difference between visual decryption and automated processing is that a person (decoder) sees the entire or almost the entire image, whereas a computer in most cases analyzes digital values for only one pixel or a small group of pixels, comparing them with the rest. A person can use different decryption signs to recognize terrain objects: size, shape, length, relative position of objects, etc., but a computer can simultaneously analyze several images in different spectral bands, and, as a rule, much faster than a person can. When carrying out any classification, some obstacles affect the quality and accuracy of the classification: CMIS-2021: The Fourth International Workshop on Computer Modeling and Intelligent Systems, April 27, 2021, Zaporizhzhia, Ukraine EMAIL: ipestova@casre.kiev.ua (I. Piestova); ak@casre.kiev.ua (A. Kozlova); a.a.andreev@casre.kiev.ua (A. Andreiev); jan.rabcan@fri.uniza.sk (J. Rabcan) ORCID: 0000-0003-2981-7826 (I. Piestova); 0000-0001-5336-237X (A. Kozlova); 0000-0002-6485-449X (A. Andreiev) ; 0000-0003- 2835-9114 (J. Rabcan) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org)  the use of supervised classification causes errors associated with the human factor, uneven samples, a discrepancy between the selected thematic classes and their spectral characteristics, etc. - this can be called a radiometric obstacle [3];  dependence of the quality of classification (detail, separability of classes) on the spatial resolution of the input images, as well as the frequent use of images of different resolutions or non-uniform spatial resolution of images (such as Sentinel-2 data, where there is a spatial resolution of 10, 20 and 60 m) - what can be called a spatial obstacle [4]. 2. Methods This study presents tools for solving the problems described above: applying a hybrid approach to classification to correct radiometry and the spatial resolution equalization of multispectral satellite images. 2.1. Hybrid approach to classification The reasonable interpretation of land cover classes could only be achieved by supervised classification since it applies expert knowledge, which describes each type of land cover. Supervised classification requires training samples of classes selected by an expert. However, those classes, as a rule, are subjective as well as expert-selected training samples are not accurate. Unlike supervised classification, unsupervised one provides objective classes, obtained by clustering. The hybrid approach to classification [5] is applied to form the interpretable and objective classes. The scheme of the described above approach to classification is shown in Figure 1. Figure 1: The scheme of the hybrid approach to classification This approach to classification implies subdividing training samples of expert-selected classes into objective clusters by unsupervised classification. After that, formed clusters are used as training samples for supervised classification. Those steps are aimed to reduce the inaccuracy and subjectiveness of the selected samples. In turn, it increases the classification accuracy in comparison with both supervised and unsupervised types of classification. 2.1.1. Training samples clustering The first step is the training samples clustering by unsupervised classification. Clustering is necessary to avoid the subjectiveness of expert-selected classes. Also, due to the high heterogeneity of some land cover classes, they overlap each other and it means that their training samples, as a rule, are mixed. This problem could be also solved by the initial training samples clustering. Basic Sequential Algorithmic Scheme (BSAS) [6] is applied as a method of unsupervised classification. The main benefit of this method among others is that the number of clusters may not be known in advance. However, instead of the number of clusters, this scheme requires two other input parameters, which are the threshold of the dissimilarity and the maximum allowed number of clusters. The first one is defined as a distance between each cluster and feature vector, which corresponds to the set of training samples. The second parameter is required not to divide the input data into a greater number of clusters than it’s defined by an expert. This parameter should be defined taking into account the computational costs of those operations and/or limit of clusters, overcoming of which will provide inadequate and not interpretable training samples subdividing. This method applies separately to the training samples of each class. As a result of this step, initial expert-selected training samples would be subdivided into objective subclasses and their heterogeneity would be reduced by transforming them into dense clusters. 2.1.2. Supervised classification The next step is the supervised classification of the study area. Support vector machine [7] is applied as a method of supervised classification for this task. This method is proved its reliability in conditions of high heterogeneity of land cover classes [8]. The required input data for this procedure are an image of the study area and training samples of each class. Subclasses obtained at the previous step are used as training samples. Therefore, each subclass is interpreted as a particular class. After this step, the input image will be classified into that number of classes, which corresponds to the number of subclasses obtained after clustering of initial training samples. In case if the task is to obtain classification, which consists only of initial classes, then merging of subclasses into initial classes is required. The subclasses’ merging defines the initial class of pixels subclass, and then the value of a pixel is converted to the value of the initial class. This procedure is performed separately for each pixel of classification obtained at the second step. 2.2. Spatial resolution equalization It is very important to carry out the spatial resolution equalization procedure when satellite images or separate image bands of different spatial resolutions are used together. A widely used method is to simply divide a pixel into subpixels according to the nearest neighbour rule [9], while the resulting subpixels retain the value of the output pixel. Although the use of the nearest neighbour rule preserves the average radiometric value of subpixels in a pixel, it does not increase the information content of the resampled bands. Another widely used method for calculating subpixel values is the application of the selected type of interpolation based on a certain number of adjacent pixels in the original image. Consider, for example, a bicubic interpolation technique [10] that slightly smoothes the image, giving a sense of more detail. However, this procedure does not keep the radiometry within the original pixel. The average value of the subpixels in pixels is different from the initial value of the input image. It is proposed to use a method [11] based on image segmentation by spectral signatures, as well as optimization of decision making when redistributing values in subpixels, taking into account both the similarity of the original spectral signatures and the spatial relationships of the topology for each type of land cover. 2.2.1. Scanning pattern The Nearest Neighbour Oversampling procedure quadruples a pixel and ultimately generates 4 identical subpixels. To improve the overall physical resolution of the image bands, but not degrade their quality, it is necessary to correctly redistribute the signals in these subpixels, while maintaining their average radiometric value. This should be done using the scanning window and taking into account the adjacent territory, that is, the nearest eight subpixels (numbered 1 .. 4 and 6 .. 9) around the current one (numbered as 5) as shown in Figure 2. Figure 2: Neighbourhood of processed subpixel (5) within scanning window (1 .. 4 and 6 .. 9) including 4 subpixels (1, 2, 4, 5) of one low‐resolution pixel When processing images, the process of determining different types of the earth's surface is necessary. This is done by spectral segmentation of the image by spectral characteristics. Fuzzy logic methods are widely used to solve such problems [12]. 2.2.2. Classes reallocating For the correct spatial redistribution of subpixels, it is necessary to analyze the topological properties of the main classes and obtain appropriate topological descriptions. Multiple valued logic (MVL) methods are used to solve the problem of changing the subpixel class according to the relationships and classes of the nearest surrounding subpixels. MVL is a type of logic in which the level of truth can be m-valued or infinite, not just binary, as in Boolean logic. MVL requires a mathematical approach to express the relationships between input logic values and the result of certain phenomena. A logical function with a value having the following form was used: f m :  n  , (1) where n is the number of multivalued variables, and the set M = {0, 1, ..., m – 1} is the set of certain truth levels. MVL is used to reclassify surface types depending on their spatial distribution, which is used in our analysis of remote sensing data because this approach will allow a fairly efficient reclassification. The pixel redistribution procedure takes into account the following topological properties of the analyzed segment:  compactness - when subpixels of one class are localized by compact inseparable groups;  orientation (linearity) - subpixels of one class are arranged linearly: horizontally, vertically, diagonally if you can determine the orientation of these lines;  texture (homogeneity) - subpixels of one or more classes are arranged in a checkerboard pattern or close to it, as well as when several different classes fall within the window and compact or linear structures are not defined. Five significant types of the relative location of subpixels of one class are defined in Figure 3. These five types represent specific shapes of subpixels, such as all columns and rows - T5, diagonals (T2), and right triangles with a certain location (T1 - the right angle at the edges, T3 - the right angle in the middle of the outer columns and rows in the matrix, T4 - the right angle in the middle of the matrix). The number of occurrences of such types can be defined as an input variable for a 3-digit logical function, which will give a decisive result for the central pixel of the class change matrix. Figure 3: The main types of mutual arrangement of subpixels. The values of the types of mutual arrangement of subpixels are consistent with the possible number of this type of arrangement in the considered block. The next step is performed in a matrix consisting of subpixels of a 3 × 3 sliding window, where the analyzed pixel is central. The classification in the scan window is checked sequentially. A logical function is used to decide on the replacement of the central subpixel class (Figure 4). According to this function, the central pixel changes if the value of the function is 2, and does not change if the value of the function is 0. Changing the class of the central pixel requires additional analysis if the value of this function is 1. Figure 4: A logical function to change the class of the central pixel depending on its location. 2.3. Radiometric‐spatial Feedback The hybrid classification method described above requires high-resolution source images. In turn, to increase the resolution of the bands of a multispectral image, stable clustering is required, which is based on the spectral signatures of the earth's covers. This leads us to the solution of using radiometric spatial feedback. This method is an iterative process described in Figure 5. Figure 5: General scheme of multispectral imagery classification local quality improvement with radiometric‐spatial feedback. The iterative process continues until the spatial resolution of the bands is increased. As soon as the increase becomes insignificant compared to the previous iteration, the process stops and the final classification is carried out. 3. Test The method of multispectral imagery classification local quality improvement with radiometric- spatial feedback in the selected study area was tested. 3.1. Study area The study area was located around the city of Novomyrhorod in Kirovohrad Oblast (administrative province) of central Ukraine (Figure 6). It encompassed agricultural, wetland, and urban landscapes which elements substantially vary in spatial characteristics. For instance, extensive and spatially homogeneous croplands have sharp delineations in a form of roads and tree lines, while wetlands and wet grassland have meandering and vague boundaries. Relatively small built-up areas extended along streets and surrounded by highly heterogeneous household plots. However, most of these elements have similar spectral characteristics especially, during the mid-summer season. For their better recognition from satellite images, the application of spectral indices strongly recommended [13-15]. Figure 6: The study area involved various urban, rural, and complex natural landscapes around Novomyrhorod, Ukraine. It is shown on the fragment of the Sentinel‐2 Multispectral Instrument (MSI) image acquired on 6 July 2020. The image represents a true‐colour composite of Red, Green, and Blue bands of 10 m spatial resolution. In this study, we focused on seven land cover types. They are artificial surfaces, croplands, tree- covered areas, grasslands, wet grasslands, wetlands, and water bodies (Table 1). Table 1 The classification scheme used in the study Land Cover Class Description Artificial surfaces Urban public and industrial built‐up areas, transport units, and construction sites Croplands Arable land, permanent crops, fallow lands, heterogeneous agricultural areas, household plots Tree‐covered areas Broadleaved and coniferous forest stands, ravine and floodplain forests, roadside tree lines, areas with tree cover more than 30% Grasslands Natural herbaceous vegetation, permanent grasslands of natural origin, pastures Wet grasslands Grassland that is periodically flooded or waterlogged by freshwater with typical plant communities of grass, sedge, and rush Wetlands Inland marshes, reed beds, riparian cane formations Water Rivers, reservoir, streams 3.2. Sentinel‐2A data and pre‐processing Cloud-free Sentinel-2A multispectral instrument (MSI) image acquired on 06 July 2020 was downloaded from the U.S. Geological Survey (USGS) archive through the EarthExplorer interface (https://earthexplorer.usgs.gov/). The images were obtained at Level 1C: top of the atmosphere (TOA) reflectance (https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi). The MSI measures the Earth's reflected radiance in 13 spectral bands from the visible near-infrared (VNIR) to shortwave infrared (SWIR) wavelength regions, with spatial resolutions from 10 to 60 m, as shown in Table 2. Table 2 Spatial and spectral resolutions of Sentinel‐2 MSI Sentinel‐2A/MSI (µm) Band Resolution (m) Band 1 (0.43–0.45) Coastal aerosol 60 Band 2 (0.46–0.52) Blue 10 Band 3 (0.54–0.58) Green 10 Band 4 (0.65–0.68) Red 10 Band 5 (0.7–0.71) Red‐edge‐1 20 Band 6 (0.73–0.75) Red‐edge‐2 20 Band 7 (0.76–0.78) Red‐edge‐3 20 Band 8 (0.78–0.90) NIR 10 Band 8A (0.85–0.87) Narrow NIR 20 Band 9 (0.93–0.95) Water vapor 60 Band 10 (1.36–1.39) SWIR/Cirrus 60 Band 11 (1.56–1.65) SWIR‐1 20 Band 12 (2.10–2.28) SWIR‐2 20 To provide atmospherically corrected images essential to calculations of spectral indices, Level 1C data was processed to Level 2A: bottom of the atmosphere reflectance (BOA). The Sen2Cor tool (https://step.esa.int/main/snap-supported-plugins/sen2cor/) from the European Space Agency (ESA) Sentinel Application Platform (SNAP) was used to perform the corrections for the Sentinel-2 image. During the processing, Sen2Cor discarded the three bands (B1, B9, and B10) that consider the effects of aerosols and water vapour on reflectance. Then, the Sentinel-2 bands acquired at 20 m data were previously resampled using the nearest neighbour method to obtain a layer stack of 10 spectral bands at 10 m. At the final stage of data pre-processing, the obtained image was resized by an area of 1500 × 1000 pixels (Figure 6) for testing the method proposed in the study. 3.3. Spectral indices use Spectral indices, being nonlinear transformations of original spectral bands, substantially enhance the classification quality of complex classes [1, 4, 16]. The red-edge (RE) is the prominent spectral feature of vegetation, including wetlands [13, 14]. The SWIR range is extensively used in many applications related to water bodies and urban surfaces [15]. Sentinel-2 MSI image of Level 2A provides the RE (B5, B6, and B7) and the SWIR (B11 and B12) ranges at 20m spatial resolution. The SI used in this study are:  (a) the normalized difference vegetation index - NDVI [17];  (b) the red-edge NDVI -RENDVI [18];  (c) the red edge ratio vegetation index – RERVI [19, 20];  (d) the normalized difference water index -NDWI [21];  (e) the modified normalized difference water index - MNDWI [22];  (f) the normalized difference moisture index - NDMI [23];  (g) the normalized difference built-up index – NDBI [24]. The formulations and the bands used to calculate the spectral indices from Sentinel-2A MSI are shown in Table 3. Table 3 The spectral indices calculated from Sentinel‐2A MSI image data in the study Vegetation Index Formulation Sentinel‐2 Bands Used NDVI RNIR  Rred B8  B4 RNIR  Rred B8  B4 RENDVI RNIR  Rred _ edge B8  B6 RNIR  Rred _ edge B8  B6 RERVI RNIR B8 Rred edge B6 NDWI Rgreen  RNIR B3  B8 Rgreen  RNIR B3  B8 MNDWI Rgreen  RSWIR B3  B11 Rgreen  RSWIR B3  B11 NDMI RNIR  RSWIR B8  B12 RNIR  RSWIR B8  B12 NDBI RSWIR  RNIR B11  B8 RSWIR  RNIR B11  B8 4. Local quality improvement Guided by the data flow diagram described in Figure 5, a two-iteration process was carried out, which included 2 hybrid classification blocks, 2 blocks of spectral bands spatial resolution enhancement, and the final classification. The results of evaluating the spatial resolution of spectral bands after the second iteration using the edge spread function are presented in Table 4 and show a steady increase of spatial resolution in bands 5-7 and 8a. It is assumed that a more stable increase of the spatial resolution in bands 11 and 12 can be achieved after attracting additional spectral signatures for reference clustering. Table 4 Digital image bidirectional Gaussian edge spread function values after the second iteration Band Basic Cubic Enhanced Band 5 1.778 × 2 2.806 2.658 Band 6 2.313 × 2 3.092 1.417 Band 7 2.157 × 2 3.047 1.759 Band 8a 2.239 × 2 3.128 1.893 Band 11 2.561 × 2 3.379 4.113 Band 12 2.043 × 2 3.009 4.125 Average 4.364 3.077 2.661 An illustration of the multispectral imagery classification local quality improvement with radiometric-spatial feedback is shown in Figure 7. Figure 7: Test example: a ‐ reference classification before the spatial resolution enhancement of spectral bands, b ‐ a subset of the area of interest, c ‐ final classification after the spatial resolution enhancement of spectral bands In the presented test section, you can see that the class divergence has increased, as a result of which there are no obvious errors in the final classification when the pixels of the artificial object class appear in the middle of the field class. All linear extended features, such as forest belts and roads, retained their shape and got rid of distortions caused by larger pixels from bands with a lower spatial resolution. 5. Conclusion The article presents a solution to a radiometric obstacle by using a hybrid approach to classification, as well as a solution to a spatial obstacle by the spatial resolution enhancement of lower-resolution spectral bands. Radiometric-spatial feedback has also been established. Further research should be aimed at developing an algorithm for determining the most appropriate clustering method for each particular class. And to increase the quality of the spatial resolution enhancement in all spectral bands, it is planned to use spectral signatures databases. 6. References [1] M. Popov, S. Michaelides, S. Stankevich, A. Kozlova, I. Piestova, M. Lubskiy, O. Titarenko, M. Svideniuk, A. Andreiev, S. Ivanov, Assessing long-term land cover changes in watershed by spatiotemporal fusion of classifications based on probability propagation: The case of Dniester river basin, Remote Sensing Applications: Society and Environment, vol. 22, 2021, 100477. doi:10.1016/j.rsase.2021.100477. [2] M. O. Popov, M. V. Тopolnytskyi, O. V. Titarenko, S. A. Stankevich, А. A. Аndreiev, Forecasting Gas and Oil Potential of Subsoil Plots via Co-analysis of Satellite, Geological, Geophysical and Geochemical Information by Means of Subjective Logic, WSEAS Transactions on Computer Research, vol. 8, 2020, pp. 90–101. [3] G.M. Foody, Impacts of ignorance on the accuracy of image classification and thematic mapping, Remote Sensing of Environment, vol. 259, 2021, 112367. doi:10.1016/j.rse.2021.112367. [4] S.I. Toure, D.A. Stow, H. Shih, J. Weeks, D. Lopez-Carr, Land cover and land use change analysis using multi-spatial resolution data and object-based image analysis, Remote Sensing of Environment, vol. 210, 2018, pp. 259–268. doi:10.1016/j.rse.2018.03.023. [5] A.A. Andreiev, Hybrid approach to classification of remote sensing data, CERes Journal, vol. 6 (2), 2020, pp. 32–37. [6] S. Theodoridis, K. Koutroumbas, Pattern Recognition, fourth edition, Elsevier, 2009, pp. 633– 635. [7] B. Tso, P. M. Mather, Classification Methods for Remotely Sensed Data, 2nd ed., CRC Press, Boca Raton, USA, 2009. [8] L. Bruzzone, B. A. Demir, A review of modern approaches to classification of remote sensing data, Land use and land cover mapping in Europe, I. Manakos, M. Braun, Eds. Springer: Dordrecht, Netherlands, 2014, pp. 127–143. [9] O. Rukundo, H. Cao, Nearest neighbor value interpolation, International Journal of Advanced Computer Science and Applications, 2012, pp. 25–30. doi:10.14569/IJACSA.2012.030405. [10] R. Keys, Cubic convolution interpolation for digital image processing, IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, 1981, pp.1153–1160. [11] I. Piestova, S. Stankevich, J. Kostolny, Multispectral imagery superresolution with logical reallocation of spectra, Proceedings of the International Conference on Information and Digital Technologies (IDT 2017), IEEE, Žilina, 2017, pp. 322–326. doi:10.1109/DT.2017.8024316. [12] E. Zaitseva, I. Piestova, J. Rabcan, P. Rusnak, Multiple-Valued and Fuzzy Logics Application to Remote Sensing Data Analysis, 26th Telecommunications Forum (TELFOR), Belgrade, Serbia, 2018, pp. 1–4, doi:10.1109/telfor.2018.8612109. [13] S. Taddeo, I. Dronova, N. Depsky, Spectral vegetation indices of wetland greenness: Responses to vegetation structure, composition, and spatial distribution, Remote Sensing of Environment, 2019, 111467. doi:10.1016/j.rse.2019.111467. [14] M. Amani, B. Salehi, S. Mahdavi, B. Brisco, Spectral analysis of wetlands using multi-source optical satellite imagery, ISPRS Journal of Photogrammetry and Remote Sensing, vol. 144, 2018, pp.119–136. doi:10.1016/j.isprsjprs.2018.07.005. [15] X. Yang, S. Zhao, X. Qin, N. Zhao, L. Lian, Mapping of Urban Surface Water Bodies from Sentinel-2 MSI Imagery at 10 m Resolution via NDWI-Based Image Sharpening, Remote Sensing, vol. 9, 2017, 596. doi:10.3390/rs9060596. [16] S. Kokhan, A. Vostokov, Using Vegetative Indices to Quantify Agricultural Crop Characteristics, Journal of Ecological Engineering, vol. 21, 2020, pp.120–127. doi:10.12911/22998993/119808. [17] T. N. Carlson, D. A. Ripley, On the relation between NDVI, fractional vegetation cover, and leaf area index, Remote Sensing of Environment, vol. 62, 1997, pp. 241–252. [18] J. Chen, C. Yang, S. Wu, Y. Chubg, A. Linton, C. Chen, Leaf chlorophyll content and surface spectral reflectance of tree species along a terrain gradient in Taiwan’s Kenting National Park, Botanical Studies, vol. 48, 2007, pp.71–77. [19] Y. Kanke, B. Tubana, M. Dalen, D. Harrell, Evaluation of red and red-edge reflectance-based vegetation indices for rice biomass and grain yield prediction models in paddy fields, Precision Agriculture, vol. 17, 2016, pp. 507–530. doi:10.1007/s11119-016-9433-1. [20] Q. Cao, Y. Miao, H. Wang, S. Huang, S. Cheng, R. Khosla, R. Jiang, Non-destructive estimation of rice plant nitrogen status with Crop Circle multispectral active canopy sensor, Field Crops Research, vol. 154, 2013, pp. 133–144. [21] S.K. McFeeters, The use of the Normalized Difference Water Index (NDWI) in the delineation of open water features, International Journal of Remote Sensing, vol. 17, 1996, pp. 1425–1432. [22] H. Xu, Modification of normalised difference water index (NDWI) to enhance open water features in remotely sensed imagery, International Journal of Remote Sensing, vol. 27, 2006, pp. 3025–3033. [23] B. C. Gao, NDWI—A normalized difference water index for remote sensing of vegetation liquid water from space, Remote Sensing of Environment, vol. 58, 1996, pp. 257–266. [24] Y. Zha, Y. Gao, S. Ni, Use of normalized difference built-up index in automatically mapping urban areas from TM imagery, International Journal of Remote Sensing, vol. 24, 2003, pp. 583– 594.