This work addresses the problem of lossy compression of volumetric images consisting of individual slices such as those produced by CT scans and MRI machines in medical imaging. We propose an extension of a single-image lossy compression method with an autoregressive context module to a sequential encoding of the volumetric slices. In particular, we remove the intra-slice autoregressive relation and instead condition the entropy model of the latent on the previous slice in the sequence. This modification alleviates the typical disadvantages of autoregressive contexts and leads to a significant increase in performance compared to encoding each slice independently. We test the proposed method on a dataset of diverse CT scan images in a setting with an emphasis on high-fidelity reconstruction required in medical imaging and show that it compares favorably against several established state-of-the-art codecs in both performance and runtime.
Medical imaging is a set of techniques and processes that
produce images of the interior of the body for the
purpose of clinical analysis, medical intervention, or visual
representation of the function of the internal organs.
Examples of common types of imaging systems are x-rays,
computed tomography (CT) scans, magnetic resonance
imaging (MRI), or ultrasound (US). Medical imaging has
become a staple tool not only for medical diagnosis and
treatment but also a crucial component of research, as it
allows researchers and physicians to establish a
knowledge base of normal anatomy and physiology to make it
possible to identify abnormalities and study the efects of
medical intervention. For these reasons, the amount of
image data produced in healthcare and medical research Figure 1: Illustrative example of a single uncompressed slice
is huge and increasing [
Image compression methods are designed for exactly
that – to enable more eficient coding of image data with
little or no loss in visual quality. The first successful im- ern image compression codecs such as BPG [
Another solution common in practice is using only accurate estimation of latent distribution since
neighborlossless compression but such methods never achieve ing slices typically have high mutual information. This
anywhere near as high compression ratios (by order of allows for higher compression ratios with no loss in the
magnitude) as lossy methods – for example the study [
The traditional approach to image compression are of image generation by the above mentioned imaging
hand-designed codecs that are implemented as hard- techniques.
coded algorithms, based on human experience and in- We show in the experimental section that this
relatuition (see Sec. 2). As with many problems in image tively simple addition outperforms considerably the
baseprocessing and computer vision in the last decade, av- line approach in which all slices are processed completely
enues are being explored on how to learn optimal codecs independently by a single image compression method.
from data. Modern research in learned image compres- Additionally, compared to processing the full volume at
sion started with the works of Toderici et al. [
The core structure of a learned method typically con- 2. Related work
sists of an autoencoder which transforms the input and
produces a latent representation of the image which will For a long time, lossy image and video compression was
constitute the bitstream. This representation is then quan- a problem solved exclusively in the traditional way by
tized so that it can be passed to an entropy coder which hand-designed methods. Some of these methods, such
losslessly converts the discrete representation to an ac- as for example H.264 [
Recently, a promising research direction is coercing coded and only then can the decoding move to the next
the reconstruction to better satisfy the expectations of pixel. As a result, a usually parallelized operation such
the human visual system even at the expense of objective as convolution cannot be computed for the whole image
(e.g. PSNR) quality. This can be achieved for example by at once but pixel by pixel in alternation with the entropy
augmenting the loss by a term that better models human coder. Another disadvantage is that the context prevents
perception (such as LPIPS [
Literature on learned compression for medical images Model details The input to our method is a sequence
is relatively scarce, this area is still dominated by more tra- of 2D slices 0, . . . , − 1 (superscripts denote slices,
ditional approaches such as compression in the wavelet subscripts pixel indices) which are processed in order.
domain [25]. Probably the closest match for the pro- The transforms to and from the latent representation
deposed method is the lossless compression of 3D volumes noted are done for each slice independently but the
by Chen et al. [26]. In our work, however, we focus on entropy model, i.e. the probability distribution ^(ˆ) of
lossy compression. Other works propose partitioning the quantized latent ˆ (hat denotes quantization
operthe image into relevant (for the diagnosis) and less rele- ation), is conditioned on the latent of the previous slice
vant regions and apply diferent compression ratios there ˆ− 1. This helps decrease the entropy of ˆ and therefore
[27]. Learned lossy compression for 2D medical images the necessary bitrate while avoiding the disadvantages
is investigated for example in [28]. of an autoregressive context model. It is done as follows:
Instead of running the context model on the currently
3. Method encoded slice in an autoregressive fashion, we run it on
the (quantized) latent ˆ− 1 of the previous slice. During
The proposed approach is based on the single image decoding, the slices are processed in the same order so
compression method by Minnen et al. [
of the latent (the entropy model). We model the distribution ^(ˆ) of the quantized • A context module that processes the image latent in latent ˆ by a per-dimension (i.e. spatial pixel and an autoregressive fashion (i.e. causally) and is also a channel) independent Laplace distribution with mean part of the entropy model parameter estimation. and scale parameters ( , ). These two parameters
Quantizer Entropy encoder Entropy decoder
Context module Entropy module for r e d o c n e r e p y H r e d o c e d r e p y H
Quantizer Entropy encoder Entropy decoder
Entropy model
for
are estimated adaptively for each image and each pixel hyper-latent = ℎ([− 1, ]). This hyper-latent
(incl. channels) of the latent by the hyper-prior and is quantized, ˆ = (), so that it can be stored in
the context module. For quantization of the latent we the bitstream. The parameters of the entropy model
use integer rounding with mean-subtraction, meaning of the quantized latent ˆ are estimated as follows. A
that the value is first ofset by the estimated mean of its context module processes the previous slice’s latent
distribution before being rounded, (image index omitted) ˆ− 1 and hyper-decoder ℎ processes the hyper-latent
ˆ. These two are concatenated and passed through an
ˆ = ⌊ − ⌉ + , (1) entropy module to produce the final entropy
paramewhere ⌊·⌉ is integer rounding. This improves perfor- ters ( , ) = ([(ˆ− 1), ℎ(ˆ)]) for each pixel
mance because then quantization doesn’t change the of the latent. With these parameters available the
lamean of the distribution, but it requires that the entropy tent can be quantized and stored in the bitstream and the
parameters of the latent are estimated before the latent is encoding proceeds to the next slice.
quantized. In particular, both of the entropy estimation During decoding, operations responsible for
estimatmodules (hyper-prior and context) must operate on non- ing the entropy model ^(ˆ) have to be executed again
quantized values , otherwise an implicit relation would because the entropy model is required by the coder to
arise. This is dificult to achieve in a single-image autore- decode ˆ from the bitstream. The hyper-latent ˆ is
degressive context model and for example the quantization coded first and since the latent of the previous slice ˆ− 1
in [
The full procedure of processing a slice is illus- decoder to finally produce the reconstructed image
trated in Fig. 2. The image is passed through an encoder ˆ = (ˆ). The decoding then proceeds to the next
, producing the latent = (). The latent is con- slice.
catenated with the latent of the previous slice, − 1, and What remains to specify is the entropy model ^(ˆ)
passed through the hyper-encoder ℎ, producing the of the hyper-latent ˆ, since that is also processed by the
entropy coder and stored in the bitstream. We model it Table 1
by per-channel Laplace distribution, meaning that each Model architecture details. conv is a Conv2D layer with kernel
channel of has its own mean and scale parameters size , stride and output channels . transpose is a
simi(, ) but those are spatially constant so that the model larly specified ConvTranspose2D. GDN and IGDN are the
is not tied to a fixed image resolution. These parameters generalized divisive normalization layer [
Details of the model architecture are concisely sum- c192 → IGDN → transpose k5 s2 c192 → IGDN → transpose marized in Tab. 1. k5 s2 c1 Hyper-encoder: conv k3 s1 c192 → PReLU → conv k5 s2 Training details In training we optimize the rate- c192 → PReLU → conv k5 s2 c192 distortion loss (image indices omitted) cH2y8p8e→r-dPeRceoLdUer→: ccoonnvvkk53ss21cc139824→ PReLU → conv k5 s2 = E∼ [− log2 ^(ˆ)] + E∼ [− log2 ^(ˆ)] Context: conv k5 s1 c384
Entropy module: conv k1 s1 c768 → PReLU → conv k1 s1 + · 2552 · E∼ [︀ ‖ − ˆ‖22]︀ , c576 → PReLU → conv k1 s1 c384
(2) where controls the rate-distortion tradeof (determines approximate target bitrate) and (), the distribution of slice in each volumetric series. This model has the same uncompressed images, is evaluated by batch averaging. encoder/decoder as the multi-slice model and the same The first two terms on the right-hand side are approxi- architecture (not weights) of the hyper-prior but does mate (theoretical) bitrates required by the entropy coder not include the context and entropy module – the hyperto encode the latents. These are used in training as an esti- decoder directly predicts the (, ) parameters of the mate of the actual bitrates because the non-diferentiable latent entropy model. In validation and testing, we use entropy coders are removed from training. this auxiliary model to compress the first slice of the vol
Our description of ^(ˆ) and ^(ˆ) so far was some- ume and then proceed sequentially with the multi-slice
what simplified. The Laplace parametric density is used model.
only as a model to conveniently parametrize the discrete The quantization operation must be approximated
durdistribution over the symbols after quantization. In the ing training because it has zero gradient almost
everyactual evaluation, however, we have to account for the where. For both the latents and hyper-latents we use
whole interval corresponding to each discrete value be- the straight-through quantization [29], which performs
cause of quantization. This is done by integrating the integer rounding in the forward pass but acts as identity
parametric density over the corresponding interval, for in the backward pass. For evaluation of the bitrate in
example the entropy models, however, we simulate quantization
^(ˆ ) = ∫︁^^− +2121 ^ (), (3)
wibsytaiyacditdnhiteteigvheeyrpu-ernori-fuodnredmceodndoevriaslaeunfedrsod(mwecittohhdeem(r−ega21ent,-s12thu)ebrtamrnaogcrteei.oTrnehaailsswhere ^ is the continuous Laplace density model in Eq. (1)) but the entropy estimation is calculated using
parametrized by (, ) corresponding to ^(ˆ ), the dis- the uniform noise simulation, which reportedly leads to
crete distribution of ˆ . In practice, this is done by using better performance [
In each training iteration we randomly sample a small
subset of consecutive slices from each image in the Dataset We trained and tested the method on the
batch and process those through the model as a small Pediatric-CT-SEG dataset of CT-scan images of various
volume. For the first slice 0 of this subset we calculate organs downloaded from the Cancer Imaging Archive
the latent 0 = (0) using an auxiliary single-image [
An example slice from the dataset is in Fig. 1. 60.0 Training We trained the model on random spatial 52.5 crops of size 256× 256 and tested it on full-resolution ]dB images. For training, we randomly chose = 3 consecu- [RN50.0 tainvde esxlipcelositaisnga tghoeosdeqcoumenptriaolmpirsoecbeesstwinege. nWteratirnaiinngedspweiethd PS47.5 PBraospeolisneed batch size 8 using the Adam optimizer [31] with an initial 45.0 VVVVCC-intra ldeeacrrneiansgedrattheeolefa1rni−ng4rfaotre 1toM1it e−ra5tifoonrsaanfotethrewrh2i0c0hkwite- 42.5 JAABPVVPEGI1GF erations. We trained a new model for 6 values of in the 40.00.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 range from 0.032 to 3.2, which on the test set results in bits per pixel 0.05 to 0.65 bits per pixel, thus achieving a compression Figure 3: Rate-distortion performance of the proposed and ratio of 25:1 to 320:1 with respect to the original images. benchmark methods on the test set of CT-scan images. Benchmark methods We compare the performance of the proposed method with a baseline learned single- AV1 video codec (essentially AV1-intra), one of today’s image compression model and a number of established top codecs from those that are readily available e.g. in traditional image compression methods. The single- browsers. In the comparison we used the libaom-av1 image baseline is a learned model with the same archi- encoder via ffmpeg configured to 12bit internal processtecture as the auxiliary model we use to compress the ing, each slice in the series is compressed individually. ifrst slice and was trained on the same train set. Compar- AV1 [32] is a video codec in terms of quality approxiison with this method shows performance gain from the mately on the level of or slightly outperforming HEVC proposed sequential processing and the context module. but unlike HEVC its use is royalty-free, it is therefore The traditional methods are a broad selection ranging arguably the best video codec readily available today from well-known and established codecs commonly used (with production-level encoders and decoders available). in practice to the state-of-the-art prototype. Such com- In our comparison, we used ffmpeg/libaom-av1 in parison therefore well positions the proposed method 12bit mode and compressed each volumetric image as a in the landscape of existing methods and gives insight video sequence consisting of the individual slices. VVC into its properties in potential use in practice. Below we [33] (H.266) is the best existing video codec nowadays briefly describe each of the methods used in the com- but its development is still ongoing and the available enparison and optionally its configuration, afterwards we coders/decoders are on the prototype level and for most provide commentary on the results summarized in Fig. 3 practical use cases prohibitively slow. Its adoption in and Tab. 2. practice, medical or otherwise, is also hindered by the
Baseline is a learned single-image compression model fact that its use is not royalty free. We used the VTM
with the same architecture as the proposed method but 18 reference implementation in 12bit mode and again
without the context and entropy module (the hyper- compressed each volumetric image as a video sequence
decoder directly predicts the entropy parameters). It consisting of the individual slices. VVC-intra [33] is the
is trained on the same train set as the proposed and uses intra mode of VVC. For single-image compression, it is
the same training schedule. JPEG [
The baseline learned method performs on the level of provided in Tab. 2, which shows the average bitrate inAVIF – the curves almost overlap. Although AVIF is un- crease/savings and PSNR quality loss/gain evaluated by doubtedly a better codec in a general setting, the learned the BD-Rate and BD-PSNR [34], respectively. We posibaseline exploits the advantage of domain specificity – it tioned VVC-intra as the reference SOTA image codec has been trained on similar CT data. BPG generally per- and compared all others to it, as they perform on the forms well on natural images where the target PSNR is test set. The table shows average bitrate savings and usually lower, but to achieve imperceptible distortion in performance gain in the middle and right column, remedical data we observed that the reconstruction PSNR spectively. Only the proposed method and VVC video should be above 55dB (for typical images with suficient achieve improvement (BD-Rate is negative and BD-PSNR structure). We suspect there is some issue with the con- is positive). ifguration of the encoder at high bitdepth processing Finally, in Tab. 3 we show the relative runtimes rebecause BPG obviously struggles with achieving high quired for processing (encode and decode) the whole PSNRs. It is no surprise that JPEG cannot compete with test set (10 volumetric images consisting of 2184 slices) the latest methods. VVC-intra does very well and outper- with respect to the proposed method (i.e. value < 1 forms AVIF by a large margin in the whole range. With means the method is faster than ours, > 1 means it is AV1 we experienced similar problems as with BPG – it slower). These runtimes are listed for bpp = .3, apapparently “saturates” at higher bitrates and struggles proximately the middle of the tested range, since the to achieve high PSNR, which is possibly again some is- traditional methods are slower at higher bitrates (the sue with the high bitdepth configuration of the encoder proposed has constant speed across the range). Here the (although we used the same encoder as for AVIF and ranking is quite diferent than in the performance. JPEG in that case it worked fine). But from comparison with and of course the baseline are the only methods faster AVIF in low to mid bitrates we can see that the sequential than the proposed, all others are slower and some of them “video” processing of the image volume is clearly benefi- quite significantly, especially the well-performing VVC cial with noticeable performance gain. This conclusion which is clearly prohibitively slow. We argue that the is further strengthened by the results of the VVC (video) video codecs are simply not fast enough for practical use. codec, which on performance alone is a clear winner of In fairness, the proposed method and the baseline run the whole comparison, outperforming all other methods on GPU (though still each slice sequentially) while the (including the proposed) by a margin in the whole range. traditional methods are CPU-only without any external
The proposed method is significantly better than the parallelization. On the other hand, our implementation is baseline (compare green and orange curves in Fig. 3), on intended only as a proof of concept and we didn’t invest average achieving almost 30% rate savings (for the same much efort into runtime optimization. For example, in quality) and 1.8dB quality increase (for the same rate). both encode and decode the encoder and decoder process It also outperforms all image codecs such as AVIF, BPG, each slice independently. In testing we really process and especially VVC-intra, which is no small feat. This them sequentially for simplicity while it is possible to is only due to the proposed sequential context because “batch” them and process in parallel (as many as the GPU the baseline alone is significantly below VVC-intra. It memory permits), which would reduce the runtime. is however still a relatively small and simple model and Contrary to usual customs, we do not provide examples and qualitative comparison of image reconstructions because due to the high reconstruction quality and similar performance of the benchmarked methods we were not able to come up with example images that demonstrate any noticeable diference – on screen all the results look identical.
But by looking at the results of VVC we see that further gains are undoubtedly possible and we hypothesize that those can be achieved for example by a stronger context module (ours is a rather simple stack of convolutions, not in any way input-adaptive) and possibly by introducing P-frames and B-frames as in video encoding. It is our hope that this work will motivate further research into such possibilities.
We presented an extension of a single-image learned com- Acknowledgments pression method to volumetric multi-slice images with an emphasis on the medical domain, where such type This project has received funding from the European of images is quite common. Although the modification Union’s Horizon 2020 research and innovation program is relatively simple and straightforward, it provides sev- under grant agreement No 965502. eral benefits – namely using a context module without introducing any problems with parallel processing in the decode and using mean-subtracted quantization. Both of References these improve performance without compromising the runtime. This we verified in the comparison with a number of established compression methods. The comparison shows: • Clear performance gain with respect to the baseline
due to the proposed sequential context. • Good performance in absolute numbers with respect
to the established codecs. • Very competitive runtimes (if GPUs are allowed).
The testing was carried out with emphasis on low-error reconstruction and even at PSNR=55dB (in most cases indistinguishable from the original) the proposed method achieves an average compression ratio of 40:1 with respect to the uncompressed original. We consider these results a solid proof of concept for compression of volumetric medical data.
Nevertheless, there are a number of things which can be improved or investigated further. For example, the used baseline model is far from SOTA so higher absolute performance can be gained by adopting one of the SOTA single-image learned methods as a backbone and extending that with the proposed context model. In this work, however, we focused more on investigating the relative gains from the sequential context rather than absolute performance. Next, in decode our method currently does not permit random access (as in “show me slice 42”), the whole volume needs to be decoded sequentially from the beginning. But this can be remedied by introducing intra-frames compressed by the single-image auxiliary method we use for the first slice. If we use a GOP size of 8 (meaning at most 8 slices need to be decoded for any chosen slice), we can estimate that the performance drop in Tab. 2 would be approximately − 11.4% → − 7.5% in rate and +0.64dB → +0.42dB in PSNR, which is still solid improvement over VVC-intra with practically usable runtimes in both encode and decode.