This paper proposes a methodology for learned compression for satellite imagery. The proposed method utilizes an image patching and stitching approach to address the high resolution of satellite images. We present rate-distortion metrics showing that this methodology outperforms JPEG2000, currently used on satellites. In addition, we demonstrate that using satellite images to train the compression model leads to superior performance compared to using non-domain-specific data. Furthermore, a detailed evaluation of the compression algorithm in a downstream classification task is conducted. The results demonstrate that 77.83% classification accuracy is still achievable for highly compressed images with a bitrate of 0.02 BPPs when the classification model is trained on images from the same compression model. The downstream classification task evaluation highlights that the performance of the classification model is highly dependent on the type of compression applied to the training data. When trained with learned compression images, the model can only classify images with an acceptable level of accuracy (>77%) if they had also undergone learned compression. Likewise, a model trained with JPEG images can only classify JPEG images with acceptable accuracy (>89%).
in learned compression can be achieved by limiting the
training data to images from this domain. For example,
As remote sensing technology develops, satellites take Tsai et al. [
Since image compression is a ubiquitous and funda- lution of less than 10 m per pixel are active, 19 of which
mental operation, it is a well-studied topic. Improve- have been launched in the last 20 years [
For a specific image domain, further enhancements 1,289 imgs train-compress 1,551 imgs 378 imgs val-compress (uncompressed)
Train
n o o C
train-class val-class val-compress (compressed)
n o iitac led fss om i a l C tortion metric and the classification downstream task. • We show that even with compression ratios as low as 0.02 BPP, a classification accuracy of 77.83% can be achieved as long as domain-specific data is utilized for training.
Recently, image compression models based on artificial
neural networks have outperformed traditional
compression methods in terms of rate and distortion. Jamil et
al. [
A.J. Hussain et al. [
JPEG, the most popular lossy image codec, is based Furthermore, Generative Adversarial Networks
on transform coding, which uses the Discrete Cosine (GANs) have also been used in image compression.
Transformation to convert an image from pixel-space to According to Jamil et al. [
More recently, Fabrice Bellard developed the BPG
format (Better Portable Graphics) that outperforms JPEG
and JPEG2000 in terms of rate and distortion [
Indradjad et al. [19] compare four diferent approaches
for satellite image compression with transform codings: a
wavelet approach by Delaunay et al. [20], bandelets [21],
JPEG 2000 [
More recently, de Oliviera et al. [
Bacchus et al. [
This section provides an overview of the methodology proposed in this work. It begins with a brief introduction to learned image compression, followed by an explanation of how the technique is adapted to suit highresolution satellite images.
This work is based on the compression model by Balle et
al. [
We directly train the model with the trade-of between = E∼ − log2 ^(^)]︀ + E∼ − log2 ^(^)]︀ ︀[ ︀[ ⏟ rate (lat⏞ents) ⏟ rate (hyper-latents) ⏞
As discussed in the introduction, a limitation of satellite to 14798 ×
14802 pixels, which causes issues for the training and inference on neural network hardware accelerators such as GPUs. Since dividing the input into patches and processing the patches independently of each other leads to visible artifacts on the borders between the patches in the stitched images, our approach resolves this issue by compressing overlapping patches.
For the stitched image, the average value of both patches
(or four patches in corners) is used for the overlapping
regions. Figure 3 illustrates the overlapping regions of a
(3)
The dataset used in this work is the Functional Map
of the world (fMoW). It was created at the John
Hopkins University Applied Physics Laboratory in Laurel,
Maryland (United States) and is publicly available at
https://github.com/fMoW/dataset [
This section provides an overview of the evaluation process and presents the results of this work.
Firstly, the utilized data set is described in detail, and how it was employed in this work. Subsequently, the data set are highlighted and discussed. Finally, the results of the downstream classification task on the compressed = + ·
Here λ controls the trade-of between rate and distortion. For the distortion the Mean Squared Error (MSE) is used, which computes the averaged pixel-wise quadratic diference between original image and distorted image:
= E∼ || − ˆ||22
The compression rate is estimated by the crossentropy between the entropy model distribution ^ and (1) images are presented.
(2) yâ
AE Bits
AD
Context Model
Φ
Entropy Parameters
N(μ, θ) Ψ
Q âz âz
AE
The partitioning of the data set used in this work is shown in Figure 1. For compression training, 1,289 images from the fMoW train set, uniformly distributed over all 63 categories, are used (train-compress). These 1,289 images are from 1,038 objects. As such, for some objects, there are multiple images taken under diferent environmental conditions.
Another set, denoted as the (val-compress), consists of 1,929 images from 1,038 objects from the fMoW validation set. The val-compress serves two purposes: one is evaluating the compression, and another is evaluating the downstream classification task. For the latter, the valcompress is split again into 1,551 images for classification training (train-class) and 378 images for classification validation (val-class).
With the parameter λ in Equation 1 the trade-of between
Figure 3: 1496 × 1496 image divided into 36 patches (256 × rate and distortion can be controlled, i.e. increasing the
256 pixels) with an overlapping region of 8 × 256 pixels be- λ leads to a smaller MSE but therefore more BPP. To
tween two patches. evaluate our model for diferent bitrates, we train the
model with diferent values for the parameter lambda. In
Figure 5 the compression results with bitrates ranging
remote sensing applications. It includes over 1 million from 0.003 BPP to 0.68 BPP are shown for an example
images of objects taken from satellites, categorized into image. The BPP of the compressed image is calculated
63 categories, such as airports, tunnel openings, zoos, and directly by dividing the file size of the encoded image by
towers. Christie et al. [
Overall the dataset contains about 628,000 training using the MSE between the compressed and the correimages and about 100,730 images for validation. The pho- sponding uncompressed images. The PSNR is defined tographs are provided as compressed JPEG- and lossless as: ︂( 255 )︂ TIFF-color images. Each object has been photographed PSNR = 10 · log10 MSE (4) in a variety of environmental settings (weather, time, season). Since this work explicitly focuses on high- As depicted on the Rate-Distortion-Curve in Figure 6, resolution satellite images, only images with a resolution the results indicate that the proposed learned compresof at least 1024 × 1024 pixels are considered. sion methodology outperforms JPEG and is also superior without Blending with Blending to the JPEG2000 compression format, which is frequently used in satellite applications.
To verify that training the compression model with domain-specific satellite images improves the downstream classification task, another compression model Each of these classification models has been used to was trained on 1,749 non-domain-specific samples from validate data sets in diferent compression scenarios: the ImageNet data-set. JPEG data sets (0.31 BPP, 0.76 BPP, 1.55 BPP), Learned
The results in Table 1 show that the domain-specific compression-compressed (LC) datasets (0.02 BPP, compression model trained with satellite images outper- 0.67 BPP, 1.07 BPP), dataset retrieved from ImageNetforms the model trained with ImageNet samples. For a trained learned Compression (0.84 BPP) and one without PSNR of 33 it yields a lower bit rate of 0.67 BPP compared compression. The result for this classification validations to 0.84 BPP achieved by the domain-agnostic model. are shown in Table 2. The columns denote the data set the classifier was trained on, the rows denote the data 5. Classification Evaluation set that was classified during validation. The results show that a classification model works best In addition to the evaluation in terms of image quality, when classifying images that were compressed with the in this section, compressed quality is represented by the same algorithm (JPEG or learned compression) as the accuracy of a classification downstream task, i.e., iden- images on which it was trained, i.e., the JPEG-trained tifying objects in satellite images. As mentioned in the classifiers classified JPEG images with accuracies over Section 4.1, compressed and uncompressed versions of 89%. In contrast, the JPEG-trained classifier only achieves the val-compress set are used, with 1,554 images used an accuracy of up to 35.21% on images compressed by for training (train-class) the classification model, and 375 learned compression. Similarly, the accuracy of the LCimages used to validate the model (val-class). trained classifiers was at least 77% when classifying LC
A dual path network1 [22] is utilized for this evalua- images (except for very low bitrate of 0.02 BPP), and no tion. For the evaluation of the classification downstream more than 39.38% when classifying JPEG-compressed images. (a) Original image
24.00 BPP (b) LC 0.003 BPP (c) LC 0.12 BPP (d) LC 0.35 BPP
(e) LC 0.63 BPP Validated with: LC 0.02 BPP LC 0.67 BPP LC 1.07 BPP LC ImageNet 0.84 BPP JPEG 0.31 BPP JPEG 0.77 BPP JPEG 1.55 BPP Uncompressed 77.83% 25.57% 26.9% 14.94% 14.41% 16.59% 15.34% 15% 10.58% 15.67% 16.63% 78.92% 11.22% 11.85% 14.32% 12.97% 12.91% 32.89% 35.21% 11.22% 89.81% 91.55% 91.55% 91.29% (f) JPEG 0.06 BPP (g) JPEG 0.14 BPP (h) JPEG 0.44 BPP (i) JPEG 0.68 BPP
The ImageNet-trained compression model demon- LC-compressed ones trained with satellite images. This strates that the training data is also crucial for the down- suggests that traditional compression methods lead to a stream classification task. It classified 78.92% of the more versatile encoding that is not as dependent on the images created by the same compression model, while specific domain. other datasets, even with high bitrate, could not attain an accuracy greater than 16.63% for any other evaluation. It fails to classify all other data sets, including the
In this work, we propose a satellite compression methodology that outperforms traditional methods (JPEG, JPEG2000) in terms of rate and PSNR. We show that images that exceed the memory of typical neural network hardware accelerators can be compressed by feeding in patch-wise parts of the image. To remove artifacts at the connection line between two patches, the connection region is smoothed by compressing overlapping patches and combining the pixels in these regions. The proposed methodology ofers superior performance compared to JPEG, and JPEG2000, commonly used for satellite imaging. We assess the efects of compression on the performance of an object classification downstream task. We demonstrate that a classification model can learn to classify images with an accuracy of 77.83% even for images compressed with a bitrate as low as 0.02 BPP. Furthermore, we show that using diferently encoded images for training and inference can deteriorate classification accuracy significantly. As such, classification models trained with JPEG images only achieve acceptable results when tested on JPEG images. Similarly, classification models trained with images compressed with learned compression models fail when tested with JPEG images.
This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 965502.