Numerical analysis of SLSSIM similarity on medical X-ray image domain Jonas Brusokas Linas Petkevičius Institute of Computer Science Institute of Computer Science Vilnius University Vilnius University Vilnius, Lithuania Vilnius, Lithuania jonas.brusokas@mif.vu.lt linas.petkevicius@mif.vu.lt Abstract—The X-ray has been adopted and used for various networks for lossless image compression, to make storing purposes including medical diagnostics. To remove noise created images and using them for calculation more efficient [15]. by new low dose X-ray imaging procedures and reduce medical Furthermore, a significant amount of research has recently image size, X-ray image reconstruction and lossless compression using deep neural networks are being researched. To enable this, been conducted on finding ways of improving X-ray imaging image similarity metrics capable of performing well on X-ray procedures, especially on automated means of disease or images must be used. In this paper, the requirements for medical abnormality detection [2], [3], [16]. X-ray similarity metrics are defined. A new similarity metric is One of the key factors to the accuracy and effectiveness proposed taking into account the quality of structures within of a deep neural network is the objective function. In image different intensity levels. An analysis is given comparing the proposed and other currently known metrics performance on reconstruction and compression image similarity metrics are real X-ray images in simulated scenarios. used as objective functions. They compare two images with Keywords—Image similarity metrics, Low dose X-ray imaging, each other and produce a scalar result denoting their degree Medical X-ray images of difference [17]. The similarity metric defines what image properties are being evaluated so the selection of an effective I. I NTRODUCTION metric is crucial. The X-ray ever since its inception has been widely adopted In this paper a similarity metric for medical X-ray images and used for various purposes. One of the key uses of X- domain is proposed and compared with other known metrics. ray imaging is in medical diagnostics. It allows for a non- In section II essential properties of X-ray images in the invasive method of diagnosing various bone structure defects, medical domain and requirements for a similarity metric are infections, arthritis, and most cancers [1]–[3]. defined. An overview of currently used metrics is made in Despite wide usage and rapid technological advances the section III and the definition of the proposed metric is made ionizing radiation emitted during X-ray imaging procedures in section IV. The analysis comparing the performance of the creates real health risks for patients [4]–[6]. Efforts have been metrics is made in section V. made to reduce the risks associated with radiation exposure by II. R EQUIREMENTS FOR MEDICAL X- RAY IMAGE creating new procedures for performing X-ray imaging. These SIMILARITY METRICS procedures (commonly referred to as low-dose) use lower Medical X-ray images are created and used to enable voltage or amperage settings to reduce the amount of radiation trained radiologists to analyse the human body, determine and emitted thus reducing the risks [7]. Unfortunately, using these diagnose various irregularities and illnesses. X-ray images can types of procedures results in images having artifacts and be done over any part of the human body and as such contain noises which can reduce diagnostic suitability [8]. There are various structures including bones, tissue, and organs [1], [2]. cases where it might not be necessary to remove the noises in Any assessment of X-ray image similarity or quality must take order to give an accurate diagnosis [9]. In most cases, however, into account properties of the images which enable them to it is imperative to remove or reduce the amount of noise on be used for accurate and reliable diagnosis. an X-ray image. Conventional approaches that use defined Through analysis of literature and working with trained properties of noise distribution do not yield satisfactory results specialists in the field of radiography several important re- [10], [11]. quirements for medical X-ray image similarity metrics have There have been some successful attempts in using deep been identified: neural networks for image reconstruction tasks, outperforming other approaches in removing various noises from images R1 The metric must detect emerging noises and artifacts [12]–[14]. There have also been attempts in using deep neural from images created with low dose X-ray procedures. As was previously stated, in most cases to enable low dose procedures, it is important to detect and remove noises ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) from the created images as they hinder diagnosis [8], [13]. 91 R2 The metric must detect perceptual geometric distortions. B. Structural similarity index (SSIM) They can appear on images during imaging, reconstruc- As a response to unsatisfactory perceived visual quality tion or decompression. From a radiologists perspective results of MSE and similar metrics a new approach was the distortions may cause difficulties or even render the proposed to construct new similarity metrics. It was based images completely unusable for diagnostics [18]. on the idea that images are highly structured and rather R3 The metric must take into account the discrepancies than evaluating pixels individually (like in MSE), groups of of images within specific ranges of brightness intensity spatially proximate and related pixels should be taken into levels. Due to the nature of the X-ray, images contain account together in order to achieve more accurate repre- various human body structures including bones, tissue, sentation of perceivable quality [17]. This is based on the organs. They are captured as areas of different brightness assumption that the human visual system is adapted to identify intensities [1]–[3]. In this paper we will refer to them various structures within field of sight and effectively spot any as sub-levels. Most image quality assessment approaches distortions affecting them [17], [22]. A well-known metric attempt to evaluate the broader perceptual view of image based on these principles is the Structural Similarity Index quality and focus on more visible areas. In medical X- (SSIM) [23]. SSIM metric calculates in account three key ray images, all structures captured in these images have features of the images: luminance changes, contrast changes significant value and greatly contribute to diagnosis [13]. and structural changes [17]. In this paper it will be defined Taking into account reconstruction or decompression ac- and used in its standard form: curacy of not only the whole picture but also structures (2µX µY + c1 )(2σXY + c2 ) within different intensity ranges is critical. SSIM∗ (X, Y ) = 2 (2) (µX + µ2Y + c1 )(σX 2 + σ2 + c ) Y 2 The proposed metric (in section IV) and following experi- ments (in section V) will take these properties into account. SSIM has seen wide usage for many image processing tasks [23]. It has served as inspiration and basis for many III. S ELECTED METRICS other similarity metrics supplementing additional features to the standard metric [24]–[28]. Research into image similarity metrics (in some cases re- C. Weighted SSIM (wSSIM) ferred to as image quality assessment) has witnessed attention and notable progress over the past decades [19]. Image quality Weighted SSIM is a general-purpose modification of the assessment is essential in fields that use image processing [20]. SSIM metric mentioned in sub-section III-B. It was created to The majority of metrics used evaluate the similarity of the be used as an objective function in Deep neural networks. It distorted image use the complete reference image [19]. In this is a composition of SSIM and traditional L1 loss. Intuitively, paper 3 selected general purpose similarity metrics which are SSIM gives the perceptual image assessment and L1 decreases potentially usable in the X-ray image domain will be discussed the metric value for more distorted images and increases for and compared. It is important to note, that for the rest of the less distorted images [29]. The authors claimed that if used paper metric definitions will be used, where metric value 1 for training, it would put more emphasis on pairs which are means the compared images are equal, and value 0, means performing worse and increase training speed and accuracy. they are completely dissimilar. In this paper L1 will be defined in its normalized form and wSSIM will be defined in the following form: A. Mean squared error (MSE) L∗1 (X, Y ) = 1 − |X − Y |L1 / (M · N ) (3) Mean squared error is considered to be one of the most wSSIM ∗ (X, Y ) = 1 − SSIM ∗ · (1 − L∗1 ) (4) simple and straight-forward similarity metrics. It is computed by averaging the intensity differences of the two compared Here X and Y are compared images, N - number of pixels images [17]. MSE is known for being quite mathematically in the images, M - maximal pixel intensity. convenient for optimization purposes. Although the metric does not correlate well with perceived visual quality [21]. IV. P ROPOSED SLSSIM METRIC Despite its flaws it still remains widely used for many image Substantial research regarding using Deep neural networks processing tasks. For the purposes of this paper, this metrics’ as a method for medical image reconstruction and processing performance will be compared to others during analysis. The is very recent [30], [31]. Although a significant amount of following normalized form of the metric will be used: metrics have been developed to suit different domains, there is no definitive similarity metric for comparing medical X-ray n 1X images. M SE ∗ (X, Y ) = 1 − (Xi − Yi )2 / M 2 (1) n i=1 As mentioned in section II a metric for medical X-ray image similarity assessment must take into account specific Here X and Y are compared images, M - maximal pixel properties of the images. Also, it has become clear from image intensity, Xi and Yi - a concrete pixel in the image, n - total quality measurement research that handcrafting and tailoring number of pixels in image. a similarity metric by combining several known metrics can 92 Fig. 1. Results with Poisson Gauss noise augmentation (left), additive Gauss noise augmentation (right) yield significantly better performance than using any single using real X-ray images from openly available medical X-ray known metric [32], [34]. image data sets. The data sets used for analysis were: In this paper a similarity metric for assessing medical X- 1) RSNA pediatric image set created by the Radiological ray image similarity is proposed. It is called the sub-level society of North America, which contains hand X-ray structural similarity index (or SLSSIM). As its name implies images [35]; it is based on the widely used structural similarity index (or 2) NIH image set created by the National Institute of Health SSIM). Its main differentiating feature is the consideration of (of the United States of America) which contains chest human body structures appearing within different sub-levels X-ray images [36]; of the image (as mentioned in requirement R3). It is achieved 3) MURA musculoskeletal image set created by Stanford by calculating SSIM values on discrete sub-levels and not the University which contains elbow, forearm, hand and entire image. This should enable more strict comparison of shoulder X-ray images [37]; the structures appearing within the sub-levels. In this paper For the analysis 3000 images were selected from the data sets sub-levels are mathematically defined as: (1000 from each) in a random order. All medical images within h iM 0 (i + 1)M 0  the data sets were gray-scale, with pixel value ranging within X (i) , Y (i) ∈ , (5) k k the interval [0; 255]. Image augmentations have been used to simulate noises Here X and Y are images, X (i) , Y (i) are sub-levels of the and distortions in the X-ray images to simulate real-life images, N - number of pixels in image, M 0 - amount of medical X-ray imaging scenarios. To analyse performance different values that a pixel can have, k - is the number of values of metrics were calculated on image pairs that contained compared discrete sub-levels. The mathematical definition of reference image – unmodified image from a data set, and SLSSIM is: v augmented image – created from reference image by applying u u Yk an augmentation. Similarity between the two images was SLSSIM = t(1 − L∗ ) · k+1 1 (SSIM ∗ (X (i) , Y (i) )) (6) evaluated using the selected and proposed similarity metrics i=0 from sections III and IV. The results were aggregated by SLSSIM is a root of a product where the members are taking the mean metric value from all the images for each normalized L1 calculated from the entire image and SSIM augmentation level. calculated from defined sub-levels. Here, L1 is used the same A. Noise detection results way as in subsection III-C, to decrease the metric value further if the image is more distorted and. Also, as all of the product As defined by requirement R1 in section II a medical X- members have values in interval [0; 1] the root is used to ray image similarity metric must detect emerging noises. Two prevent a steep downward gradient of the metric. We will be image augmentations simulating real life scenarios have been using 8 sub-levels (k = 8) for metrics performance analysis. selected to generate augmented images for testing: • Poisson-Gauss noise augmentation PG(a, b) – represents V. M ETRICS ANALYSIS artifacts and noises emerging in images obtained via Analysis has been carried out to evaluate performance of the computed tomography X-ray imaging when using X-ray selected metrics in real-life scenarios. Tests were conducted sensors in low dosage configuration [8]. The augmented 93 Fig. 2. Results with rotation augmentation (left), Y-axis translation augmentation (right) image was generated using parameters: a = 31 and B. Geometric distortion detection results b = 0.05k, k ∈ [0; 20], k ∈ Z. Higher b values result As defined by requirement R2 in section II a medical X-ray in more noise in the image. There are confirmed results image similarity metric must also detect perceptual geometric stating that noises generated with parameter b ≈ 0.1 can distortions. Two widely used image augmentations have been occur in low dose imaging [8]. selected to simulate geometric transformations and generate • Additive Gauss noise augmentation N (µ, σ) – repre- augmented images for testing: sents general noises and artifacts that can appear in X- • Translation augmentation – used to evaluate general met- ray images. Some research points to its applicability in ric robustness against translation (shifting of the image). simulating noises emerging from low dose imaging as Theoretically, translation can occur during reconstruction well [38]. The augmented image was generated using or decompression. The augmented image was generated parameters: µ = 0, σ ∈ [0; 20], σ ∈ Z. Higher σ values by translating the reference image on the Y axis by y result in more noise in the image. Visually, the noise pixels, where values y = {−30, −28, ..., 26, 28, 30} becomes noticeable at σ ≈ 5. • Rotation augmentation – also used to evaluate general metric robustness. The augmented image was generated 1) Poisson Gauss noise detection results: On the left side of by rotating the reference image by r degrees, where figure 1 similarity metrics ability to detect emerging Poisson- values r = {−30, −28, ..., 26, 28, 30} Gauss noise is observed. Metrics L∗1 , M SE ∗ , wSSIM ∗ do 1) Rotation augmentation detection results: On the left not detect the noise even when the noise parameter b is high side of figure 2 similarity metrics ability to detect rotations (images are highly disrupted). Whereas, SSIM ∗ metric and is observed. As an emerging pattern, metrics L∗1 , M SE ∗ , the proposed metric SLSSIM perform well. When noise wSSIM ∗ do not detect the distortions even when the degree parameter b = 0.1, metric values SLSSIM ≈ 0.78 and of rotation is very high. SSIM ∗ and SLSSIM still perform SSIM ∗ ≈ 0.66. These results are acceptable, as visually the well, with SSIM ∗ once again being slightly more sensitive noise should still allow for accurate diagnosis. to the level of distortion. 2) Additive Gauss noise detection results: On the right side 2) Translation augmentation detection results: On the right of figure 1 similarity metrics performance against emerging side of figure 1 similarity metrics performance in detecting additive Gauss noise is witnessed. Similarly to the previous translations is observed. Once again, metrics L∗1 , M SE ∗ , results, metrics L∗1 , M SE ∗ , wSSIM ∗ do not detect the noise wSSIM ∗ do not detect distortions well and metrics SSIM ∗ , well, the resulting values do not drop below 0.94. However, SLSSIM ∗ do so. in this case the difference between the proposed SLSSIM metric and SSIM ∗ is higher. When noise parameter σ = 5, C. Analysis results metric values SLSSIM ≈ 0.84 and SSIM ∗ ≈ 0.76. As observed in the tests L∗1 , M SE ∗ , wSSIM ∗ metrics Although these values are still acceptable, the rate at which are not suitable for accurately detecting noises or geometric SLSSIM value decreases as the noise level rises is not as transformations in medical X-ray images. While tests show representative of the added disruptions as the SSIM ∗ . It can that SSIM ∗ is more sensitive to additive Gauss noise, but be stated that SSIM is slightly more accurate in detecting SLSSIM has an advantage in being the only metric that additive Gauss noise. specifically measures discrepancies in sub-levels of the image. 94 Fig. 3. Sample chest X-ray image from NIH data set (left), same image distorted with Poisson Gauss noise (right) Fig. 4. Sample wrist X-ray image from MURA data set (left), image distorted with Poisson Gauss noise (center), visualisation of sub-level range [64; 127] of distorted image (right) Both SSIM ∗ and SLSSIM metrics are potentially suitable seen on the right. It simulates the effect that low dose imaging for use with medical X-ray images. would have on the image [8]. The metric values calculated from reference and augmented images are: M SE ∗ = 0.99, VI. S AMPLES OF METRICS BEHAVIOUR ON X-R AY IMAGES L∗1 = 0.93, SSIM ∗ = 0.10, wSSIM ∗ = 0.93 and Sample images from the data sets have been displayed to SLSSIM = 0.36. better visualise the impact that augmentations have on the Similarly, in figure 4 a reference X-Ray image from the images. Also, metric values are given for each image pair. MURA musculoskeletal X-ray image data set can be seen on In figure 3 a reference X-Ray image from NIH chest X-ray the left. An augmented image distorted by Poisson-Gaussian image data set can be seen on the left. An augmented image noise, with same parameters as before can be seen in the distorted by Poisson-Gaussian noise, a = 31, b = 0.08 can be center. The image on the right represents a visualisation of 95 the augmented image sub-level of intensity range [64; 127]. In [12] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, this particular case, this sub-level represents the bone structure “Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion,” J. Mach. Learn. from the image, with tissue and other matter being barely Res., vol. 11, pp. 3371–3408, 2010. represented. The metric values calculated from reference and [13] D. Wu, K. Kim, G. El Fakhri, and Q. Li, “A Cascaded Convolutional augmented images are: M SE ∗ = 0.98, L∗1 = 0.88, SSIM ∗ = Neural Network for X-ray Low-dose CT Image Denoising,” May 2017. [14] J. Xie, L. Xu, and E. Chen, “Image Denoising and Inpainting with 0.08, wSSIM ∗ = 0.89 and SLSSIM = 0.27. It is important Deep Neural Networks,” pp. 341–349, 2012. to note that the metrics values taken from images seen in [15] G. Toderici et al., “Full Resolution Image Compression with Recurrent figures 3 and 4 correspond with the results of the analysis Neural Networks,” Aug. 2016. in section V. [16] Q. Ke et al., “A neuro-heuristic approach for recognition of lung diseases from X-ray images,” Expert Syst. Appl., vol. 126, pp. 218–232, Jul. 2019. VII. C ONCLUSIONS [17] Z. Wang, A. C. Bovik, H. R. Sheikh, S. Member, E. P. Simoncelli, and S. Member, “Image Quality Assessment: From Error Visibility to Although there are many general purpose image similarity Structural Similarity,” vol. 13, no. 4, pp. 1–14, 2004. metrics used in various domains, only a few of them are [18] T. Černius, “Image reconstruction of x-ray images using deep learning,” Vilnius University, 2018, (In Lithuanian). suitable for use in comparing medical X-ray images. Based [19] K. Xu, X. Liu, H. Cai, and Z. Gao, “Full-reference image quality on the analysis performed on multiple medical X-ray image assessment-based B-mode ultrasound image similarity measure,” Jan. data sets and using different augmentations, L∗1 , M SE ∗ , 2017. [20] D. M. Chandler, “Seven Challenges in Image Quality Assessment: Past, wSSIM ∗ metrics are ineffective in this domain as metric Present, and Future Research,” ISRN Signal Process., vol. 2013, pp. values never drop below 0.8 even when significant distortions 1–53, 2013. are applied. SSIM ∗ and the proposed SLSSIM can be used [21] A. B. Watson and Bernd, Digital images and human vision. MIT Press, to effectively compare X-ray images as both metrics’ value 1993. [22] Z. Wang, A. C. Bovik, and L. Lu, “Why is image quality assessment is much lower when comparing heavily distorted images, in so difficult?,” in IEEE International Conference on Acoustics Speech some cases going below 0.3. Both metrics perform similarly and Signal Processing, 2002, p. IV-3313-IV-3316. in tested cases. Unlike SSIM ∗ , SLSSIM takes into account [23] W. Lin and C.-C. Jay Kuo, “Perceptual visual quality metrics: A survey,” J. Vis. Commun. Image Represent., vol. 22, no. 4, pp. 297–312, differences between human body structures in the images, May 2011. which should allow diagnosticians to make more accurate [24] F. Gao, Y. Wang, P. Li, M. Tan, J. Yu, and Y. Zhu, “DeepSim: Deep diagnoses. The proposed SLSSIM can be used as a basis of similarity for image quality assessment,” Neurocomputing, vol. 257, pp. 104–114, Sep. 2017. an objective function in medical X-ray image reconstruction [25] R. Reisenhofer, S. Bosse, G. Kutyniok, and T. Wiegand, “A Haar or compression tasks. Wavelet-Based Perceptual Similarity Index for Image Quality Assess- ment,” Jul. 2016. R EFERENCES [26] L. Zhang and H. Li, “SR-SIM: A fast and high performance IQA index based on spectral residual,” in 2012 19th IEEE International Conference on Image Processing, 2012, pp. 1473–1476. [1] U. Eglė, “Diagnostic X-ray hardware,” 2011, (In Lithuanian). [27] W. Xue, L. Zhang, X. Mou, and A. C. Bovik, “Gradient Magnitude [2] M. Woźniak and D. Połap, “Bio-inspired methods modeled for respi- Similarity Deviation: A Highly Efficient Perceptual Image Quality ratory disease detection from medical images,” Swarm Evol. Comput., Index,” IEEE Trans. Image Process., vol. 23, no. 2, pp. 684–695, Feb. vol. 41, pp. 69–96, Aug. 2018. 2014. [3] Wozniak, M., Polap, D., Kosmider, L., Napoli, C., and Tramontana, E., “A novel approach toward X-ray images classifier”. In 2015 IEEE [28] Lin Zhang, Lei Zhang, Xuanqin Mou, and D. Zhang, “FSIM: A Feature Symposium Series on Computational Intelligence, pp. 1635-1641, Similarity Index for Image Quality Assessment,” IEEE Trans. Image IEEE, 2015. Process., vol. 20, no. 8, pp. 2378–2386, Aug. 2011. [4] B. V. Daga, V. R. Shah, and S. V. Daga, Radiodiagnosis, nuclear [29] N. Johnston et al., “Improved Lossy Image Compression with Priming medicine, radiotherapy and radiation oncology. Jaypee Brothers Med- and Spatially Adaptive Bit Rates for Recurrent Networks,” Mar. 2017. ical Publishers, 2013. [30] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Net- [5] A. Berrington de González et al., “Projected Cancer Risks From works for Biomedical Image Segmentation,” May 2015. Computed Tomographic Scans Performed in the United States in 2007,” [31] G. Wang, J. C. Ye, K. Mueller, and J. A. Fessler, “Image Reconstruction Arch. Intern. Med., vol. 169, no. 22, p. 2071, Dec. 2009. is a New Frontier of Machine Learning,” IEEE Trans. Med. Imaging, [6] Y. Y. Kim, H. J. Shin, M. J. Kim, and M.-J. Lee, “Comparison of vol. 37, no. 6, pp. 1289–1296, Jun. 2018. effective radiation doses from X-ray, CT, and PET/CT in pediatric [32] S. Bosse, D. Maniry, K.-R. Muller, T. Wiegand, and W. Samek, “Deep patients with neuroblastoma using a dose monitoring program.,” Diagn. Neural Networks for No-Reference and Full-Reference Image Quality Interv. Radiol., vol. 22, no. 4, pp. 390–4, 2016. Assessment,” IEEE Trans. Image Process., vol. 27, no. 1, pp. 206–219, [7] S. P. Raman, M. Mahesh, R. V. Blasko, and E. K. Fishman, “CT scan Jan. 2018. parameters and radiation dose: Practical advice for radiologists,” J. Am. [33] Wozniak, M., Polap, D., Nowicki, R. K., Napoli, C., Pappalardo, Coll. Radiol., vol. 10, no. 11, pp. 840–846, 2013. G., and Tramontana, E., “Novel approach toward medical signals [8] S. Lee, M. S. Lee, and M. G. Kang, “Poisson–Gaussian Noise Analysis classifier”. In 2015 International Joint Conference on Neural Networks and Estimation for Low-Dose X-ray Images in the NSCT Domain,” (IJCNN), pp. 1-7, IEEE, 2015. Sensors, vol. 18, no. 4, p. 1019, Mar. 2018. [34] V. V. Lukin, N. N. Ponomarenko, O. I. Ieremeiev, K. O. Egiazarian, [9] A. Neverauskiene et al., “Image based simulation of the low dose com- and J. Astola, “Combining full-reference image visual quality metrics puted tomography images suggests 13 mAs 120 kV suitability for non- by neural network,” 2015, p. 93940K. syndromic craniosynostosis diagnosis without iterative reconstruction [35] “RSNA Bone Age data set from Kaggle.” [Online]. Available: algorithms,” Eur. J. Radiol., vol. 105, pp. 168–174, Aug. 2018. https://www.kaggle.com/kmader/rsna-bone-age. [Accessed: 01-Mar- [10] Jiang Hsieh, Computed Tomography: Principles, Design, Artifacts, and 2019]. Recent Advances, Second Edition, 2009. [36] “NIH Chest X-rays data set from Kaggle.” [Online]. Avail- [11] L. L. Geyer et al., “State of the Art: Iterative CT Reconstruction able: https://www.kaggle.com/nih-chest-xrays/data. [Accessed: 01- Techniques,” Radiology, vol. 276, no. 2, pp. 339–357, Aug. 2015. Mar-2019]. 96 [37] “Stanford University MURA Dataset: Towards Radiologist-Level Ab- normality Detection in Musculoskeletal Radiographs.” [Online]. Avail- able: https://stanfordmlgroup.github.io/competitions/mura/. [Accessed: 01-Mar-2019]. [38] Hanying Li, T. Toth, S. McOlash, Jiang Hsieh, and N. Bromberg, “Simulating low dose CT scans by noise addition,” in 2002 IEEE Nuclear Science Symposium Conference Record, 2002, vol. 3, pp. 1832–1834. 97