Towards Automated Hyperspectral Document Image Analysis Zohaib Khan, Faisal Shafait and Ajmal Mian School of Computer Science and Software Engineering The University of Western Australia, 35 Stirling Highway, CRAWLEY, 6009 Email: zohaib@csse.uwa.edu.au, faisal.shafait@uwa.edu.au, ajmal.mian@uwa.edu.au Abstract—Hyperspectral imaging and analysis refers to the spectral image. An image acquired at more than three specific capture and understanding of image content in multiple spectral wavelengths in a band is referred to as a Multispectral Image. channels. Satellite and airborne hyperspectral imaging has been Generally, multispectral imaging sensors acquire more than the focus of research in remote sensing applications since nearly three spectral bands. An image with a higher spectral resolu- the past three decades. Recent use of ground-based hyperspectral tion or more number of bands is regarded as a Hyperspectral imaging has found immense interest in areas such as medical imaging, art and archaeology, and computer vision. In this paper, Image. There is no clear demarcation with regards to the we make an attempt to draw closer the forensic community and number of spectral bands/resolution between multispectral and image analysis community towards automated forensic document hyperspectral images. However, hyperspectral sensors may examination. We believe that it has a huge potential to solve acquire a few dozen to several hundred spectral measurements various challenging document image analysis problems, especially per scene point. For example, the AVIRIS (Airborne Visi- in the forensic document examination domain. We present the ble/Infrared Imaging Spectrometer) of NASA has 224 bands use of hyperspectral imaging for ink mismatch detection in in 400-2500nm range [2]. handwritten notes as a sample application. Overall, this paper provides an overview of the applications of hyperspectral imaging with focus on solving pattern recognition problems. We hope that this work will pave the way for exploring its true potential in the During the past several years hyperspectral imaging has document analysis research field. found its utility in various ground-based applications. The use of hyperspectral imaging in archeological artifacts restoration Keywords—Multispectral imaging, Hyperspectral imaging, Hy- has shown promising results. It is now possible to read perspectral document analysis, forensic document examination, ink the old illegible historical manuscripts by restoration using mismatch detection hyperspectral imaging [3]. This was a fairly difficult task for a naked eye due to its limited capability, restricted to the visible I. I NTRODUCTION spectral range. Similarly, hyperspectral imaging has also been applied to the task of material discrimination. This is because Human eye exhibits a trichromatic vision. This is due to the of the physical property of a material to reflect a specific range presence of three types of photo-receptors called Cones that of wavelengths giving it a spectral signature which can be are sensitive to different wavelength ranges in the visible range used for material identification [4]. The greatest advantage of of the electromagnetic spectrum [1]. Conventional imaging hyperspectral imaging in such applications is that it is non- sensors and displays (like cameras, scanners and monitors) are invasive and thus does not affect the material under analysis developed to match the response of the trichromatic human compared to other invasive techniques which inherently affect vision so that they deliver the same perception of the image the material under observation. as in a real scene. This is why an RGB image constitutes three spectral measurements per pixel. Most of the computer vision applications do not make use of the spectral information and directly employ grayscale images for image understanding. Despite the success of hyperspectral imaging in solving There is evidence that machine vision tasks can take the advan- various challenging computer vision problems in recent years, tage of image acquisition in a wider range of electromagnetic its use in the document image analysis research has remained spectrum capturing more information in a scene compared largely unexplored. In this paper, we intend to draw the to only RGB data. Hyperspectral imaging captures spectral attention of the document analysis and forensics community reflectance information for each pixel in a wide spectral range. towards this promising technology. We believe that there is It also provides selectivity in the choice of frequency bands. a huge potential in hyperspectral imaging to solve various Satellite based hyperspectral imaging sensors have long been challenging document image analysis problems, especially in used for astronomical and remote sensing applications. Due to the forensic document examination domain. First, we present the high cost and complexity of these hyperspectral imaging in Section II a brief survey on the applications of hyperspectral sensors, various techniques have been proposed in the literature imaging in the field of pattern recognition. Then, some of to utilize conventional imaging systems combined with a few our recent work on forensic document examination using off-the-shelf optical devices for hyperspectral imaging. hyperspectral imaging is discussed in Section III. The paper is concluded with some hints about directions for future research Strictly speaking, an RGB image is a three channel multi- in Section IV. Fig. 1. A hyperspectral image is represented as a 3D cube (shown in pseudo-colors in center). Each slice of the cube along the spectral dimension Sλ is regarded as a channel or a band. Point spectrum on the spectral cube at the (x, y) spatial location (left). An RGB image and a grayscale image rendered from the hyperspectral cube (right). II. H YPERSPECTRAL I MAGING AND A PPLICATIONS et al. [5] or Hao et al. [6]. The underlying principle of a mul- tispectral palmprint imaging device is to use a monochromatic A hyperspectral image has three dimensions: two spatial camera with illumination sources of different colors. Images (Sx and Sy ) and one spectral (Sλ ) (see Figure 1). The of a palm are sequentially captured under each illumination hyperspectral data can be represented in the form of a Spectral within a fraction of a second. Cube. Similarly, a hyperspectral video has four dimensions – two spatial dimensions (Sx and Sy ), a spectral dimension (Sλ ) Multispectral palmprint recognition system of Han et al. [5] and a temporal dimension (t). The hyperspectral video can captured images under four different illuminations (red, green, be thought of as a series of Spectral Cubes along temporal blue and infrared). The first two bands (blue and green) dimension. Hyperspectral imaging has been applied in various generally showed only the line structure, the red band showed areas, some of which are listed in Table I. In the following, both line and vein structures, whereas the infrared band showed we provide a brief survey of the applications of hyperspectral only the vein structure. These images can be fused and fea- imaging in pattern recognition. The scope of our survey is tures extracted for subsequent matching and recognition. The limited to the multispectral and hyperspectral imaging systems contact-free imaging system of Hao et al. [6] acquires multi- used in ground-based computer vision applications. Therefore, spectral images of a palm under six different illuminations. The high cost and complex sensors for remote sensing, astronomy, contact-free nature of the system offers more user acceptability and other geo-spatial applications are excluded from the dis- while maintaining a reasonable accuracy. Experiments show cussion. that pixel level fusion of multispectral palmprints has better TABLE I. A PPLICATIONS OF HYPERSPECTRAL IMAGING IN DIFFERENT recognition performance compared to monochromatic images. AREAS . The accuracy achievable by multispectral palmprints is much higher compared to traditional monochromatic systems. Areas Applications Art and Archeology Analysis of works of art, historical artifact restoration Fingerprints are established as one of the most reliable bio- Medical Imaging MRI imaging, microscopy, biotechnology metrics and are in common use around the world. Fingerprints Military Surveillance, access control can yield even more robust features when captured under a Pattern Recognition Material identification, biometrics Remote Sensing Crop monitoring, mineralogy, water observation multispectral sensor. Rowe et al. [7] developed a multispectral imaging sensor for fingerprint imaging. The system comprised of illumination source of multiple wavelengths (400, 445, 500, 574, 610 and 660nm) and a monochrome CCD of 640x480 A. Biometrics Applications resolution. They showed that MSI sensors are less affected The bulk of biometric recognition research revolves around by moisture content of skin which is of critical significance monochromatic imaging. Recently, different biometric modali- compared to the traditional sensors. Recognition based on ties have taken advantage of hyperspectral imaging for reliable multispectral fingerprints outperformed standard fingerprint and improved recognition. The images can cover visible, imaging. infrared, or a combination of both ranges of the electromag- Face recognition has an immense value in human iden- netic spectrum (see Figure 2). We briefly discuss the recent tification and surveillance. The spectral response of human work in palmprint, face, fingerprint, and iris recognition using skin is a distinct feature which is largely invariant to the pose hyperspectral imaging. and expression [8] variation. Moreover, multispectral images Palmprints have emerged as a popular choice for human of faces are less susceptible to variations in illumination access control and identification. Interestingly, palmprints have sources and their directions [9]. Multispectral face recognition even more to offer when imaged under different spectral systems generally use a monochromatic camera coupled with ranges. The line pattern is captured in the visible range a Liquid Crystal Tunable Filter (LCTF) in the visible and/or while the vein pattern becomes apparent in the near infrared near-infrared range. A multispectral image is captured by range. Both line and vein information can be captured using a electronically tuning the filter to the desired wavelengths and multispectral imaging system such as those developed by Han acquiring images in a sequence. 100 nm 1000 nm 10 nm 100 um 10 um Mid−Wave IR Short−Wave IR Long−Wave IR Near IR Far UV Middle UV Near UV Visible 200 nm 300 nm 400 nm 700 nm 10 nm 1.4 um 3 um 5 um 7 um 14 um Fig. 2. The electromagnetic spectrum. Iris is another unique biometric used for person authenti- cheaper and more efficient than traditional analytical chemistry cation. Boyce et al. [10] explored multispectral iris imaging methods [16]. For that purpose, near-infrared spectrometers are in the visible electromagnetic spectrum and compared it to used that measure the spectrum of light transmitted through a the near-infrared in a conventional iris imaging systems. The sample of minced pork meat. use of multispectral information for iris enhancement and segmentation resulted in improved recognition performance. Last but not least, multispectral imaging has also important applications in defense and security. For instance, Alouini [17] showed that multispectral polarimetric imaging significantly B. Material Identification enhances the performance of target detection and discrimina- Naturally existing materials show a characteristic spectral tion. response to incident light. This property of a material can distinguish it from other materials. The use of multispectral III. F ORENSIC D OCUMENT E XAMINATION USING techniques for imaging the works of arts like paintings allows H YPERSPECTRAL I MAGING segmentation and classification of painted parts. This is based on the pigment physical properties and their chemical com- Hyperspectral imaging (HSI) has recently emerged as an position [3]. Pelagotti et al. [11] used multispectral imaging efficient non-destructive tool for detection, enhancement [18], for analysis of paintings. They collected multispectral images comparison and identification of forensic traces [19]. Such of a painting in UV, Visible and Near IR band. It was systems have a huge potential for aiding forensic document possible to differentiate among different color pigments which examiners in various tasks. Brauns et al. [20] developed a appear similar to the naked eye based on spectral reflectance hyperspectral imaging system to detect forgery in potentially information. fraudulent documents in a non-destructive manner. A more so- phisticated hyperspectral imaging system was developed at the Gregoris et al. [12] exploited the characteristic reflectance National Archives of Netherlands for the analysis of historical of ice in the infrared band to detect ice on various surfaces documents in archives and libraries [21]. The system provided which is difficult to inspect manually. The developed prototype high spatial and spectral resolution from near-UV through called MD Robotics’ Spectral Camera system could determine visible to near IR range. The only limitation of the system the type, level and location of the ice contamination on a was its extremely slow acquisition time (about 15 minutes) surface. The prototype system was able to estimate thickness [22]. Other commercial hyperspectral imaging systems from of ice (<0.5mm) in relation to the measured spectral contrast. Foster & Freeman [23] and ChemImage [24] also allow manual Such system may be of good utility for aircraft/space shuttle comparison of writing ink samples. Hammond [25] used visual ice contamination inspection and road condition monitoring in comparison in Lab color mode for differentiating different snow conditions. black inks. Such manual analysis of inks cannot establish the Multispectral imaging has critical importance in magnetic presence of different inks with certainty, because of inherent resonance imaging. Multispectral magnetic resonance imagery human error. Here we will demonstrate a promising application of brain is in wide use in medical science. Various tissue of hyper-spectral imaging for automated writing inks mismatch types of the brain are distinguishable by virtue of multispectral detection that we have recently proposed [26]. The work is imaging which aids in medical diagnosis [13]. based on the assumption that same inks exhibit similar spectral responses whereas different inks show dissimilarity in their Clemmensen et al. [14] used multispectral imaging to spectra. The phenomenon is illustrated in Figure 3. We assume estimate the moisture content of sand used in concrete. It is a that the spectral responses of the inks are independent of the very useful technique for non-destructive in-vivo examination writing styles of different subjects. of freshly laid concrete. A total of nine spectral bands was acquired in both visual and near infrared range. Zawada et Using our hyperspectral imaging setup (see [26] for de- al. [15] proposed a novel underwater multispectral imaging tails), a database comprising of 70 hyperspectral images of system named LUMIS (Low light level Underwater Multispec- a hand-written note in 10 different inks by 7 subjects was tral Imaging System) and demonstrated its use in study of collected1 . All subjects were instructed to write the same phytoplankton and bleaching experiments. sentence, once in each ink on a white paper. The pens included Spectrometry techniques are also widely used to identify 1 UWA Writing Ink Hyperspectral Image Database the fat content in pork meat, because it has proved significantly http://www.csse.uwa.edu.au/%7Eajmal/databases.html RGB 460nm 520nm 580nm 640nm 700nm Blue Ink Black Ink 1 1 0.8 0.8 Accuracy Accuracy 0.6 0.6 0.4 0.4 Fig. 3. The above images highlight the discrimination of inks offered 0.2 0.2 RGB RGB by hyperspectral images. We show a selected number of bands at specific HSI HSI wavelengths for two different blue inks (for the word ‘fox’). Notice that 0 C12 C13 C14 C15 C23 C24 C25 C34 C35 C45 0 C12 C13 C14 C15 C23 C24 C25 C34 C35 C45 only the pixels belonging to the writing pixels are shown and the pixels of the background are masked out. A closer look allows one to appreciate that hyperspectral imaging captures subtle differences in the inks, which are Fig. 5. Comparison of RGB and HSI image based segmentation accuracy. enhanced, especially at higher wavelengths. HSI significantly improves over RGB in most of the ink 5 varieties of blue ink and 5 varieties of blank ink pens. It was combinations. This results in most accurate clustering of ink ensured that the pens came from different manufacturers while combinations C12 , C14 , C12 , C25 , C35 and C45 . In case of the inks still appeared visually similar. Then, we produced black inks, ink 1 is highly distinguished from all other inks mixed writing ink images from single ink notes by joining resulting in the most accurate clustering for all combinations equally sized image portions from two inks written by the C1j . However, it can be seen that for a few combinations, same subject. This made roughly the same proportion of the HSI does not show a remarkable improvement. Instead, in two inks under question. some cases, it is less accurate compared to RGB. These results The mixed-ink images were pre-processed (binariza- encouraged us to further look at HSI in detail in order to tion [27] followed by spectral response normalization) and then take advantage of the most informative bands. The results of fed to the k-means clustering algorithm with a fixed value of different feature (band) selection methods for this problem are k = 2. Finally, based on the output of clustering, segmentation detailed in [26]. Overall, the results showed that use of a few accuracy was computed as selected bands further improved discrimination between most of the ink combinations. True Positives Accuracy = We now present some qualitative results on segmentation True Positives + False Positives + False Negatives of blue and black ink combinations. The original images of The segmentation accuracy is averaged over seven samples for a combination of two blue inks (C34 ) and black inks (C45 ) each ink combination Cij . It is important to note that according are shown are in Figure 6. RGB images are shown here for to this evaluation metric, the accuracy of a random guess (in a better visual appearance. The ground truth images are labeled two class problem) will be 1/3. This is different from common in pseudo-colors, where green pixels represent the first ink and classification accuracy metrics where the accuracy of a random red pixels represent the second ink. guess is 1/2. This is because our chosen metric additionally penalizes false negatives which are useful to quantify in a The clustering based on RGB images fails to group similar segmentation problem. ink pixels into the same clusters. A closer look reveals that all of the ink pixels are falsely grouped into one cluster, whereas Blue Ink Black Ink most of the boundary pixels are grouped into the other cluster. Ink 1 Ink 1 Mean Normalized Spectra 0.22 0.22 This implies that typical RGB imaging is not sufficient to Mean Normalized Spectra Ink 2 Ink 2 Ink 3 Ink 3 0.2 Ink 4 0.2 Ink 4 Ink 5 discriminate inks that appear visually similar to each other. Ink 5 0.18 0.18 On the other hand, segmentation based on HSI is much more effective compared to RGB. It can be seen that the majority 0.16 0.16 of the ink pixels are correctly grouped in HSI in accordance 0.14 0.14 with the ground truth segmentation. Note that the k-means clustering algorithm used in this work is rather basic. The 400 500 600 700 400 500 600 700 Wavelength (nm) Wavelength (nm) use of more advance clustering algorithms has the potential of further improving the accuracy of ink segmentation. Fig. 4. Spectra of the blue and black inks under analysis. Note that at some ranges the ink spectra are more distinguished than others. IV. C ONCLUSION AND O UTLOOK Figure 4 shows the average normalized spectra of all blue This paper presented an overview about different appli- and black inks, respectively. It was achieved by computing the cations of hyperspectral imaging in pattern recognition. We average of the spectral responses of each ink over all samples also demonstrated a sample application of HSI in document in the database. It can be observed that the spectra of the inks image analysis, where it was possible to discriminate between are distinguished at different ranges in the visible spectrum. A two visually similar inks using hyperspectral images of the close analysis of variability of the ink spectra in these ranges documents. This is the first reported work on using auto- reveals that most of the differences are present in the high- matic document image analysis methods in combination with visible range, followed by mid-visible and low-visible ranges. hyperspectral imaging to address forensically relevant issues We now inspect how hyperspectral information can be in questioned document examination. In future, it will be beneficial in discrimination of inks. We compare the segmen- interesting to see whether spectral imaging can aid in writer tation accuracy of HSI with RGB in Figure 5. As expected, identification. Since it is possible to identify hand writings Original Image Ground Truth Result (RGB) Result (HSI) Fig. 6. Example test images. For a visual comparison of RGB and HSI mismatch detection accuracy, we purposefully selected two hard cases. by the texture [28] or ink-deposition traces [29], a promising [11] A. Pelagotti, A. Del Mastio, A. De Rosa, and A. Piva, “Multispectral research direction would be to investigate whether these feeble imaging of paintings,” IEEE Signal Processing Magazine, vol. 25, no. 4, variations in ink strokes are reflected in the spectral response of pp. 27–36, 2008. the inks. In addition, ink or document aging is a phenomenon [12] D. Gregoris, S. Yu, and F. Teti, “Multispectral imaging of ice,” in Proc. Canadian Conference on Electrical and Computer Engineering, vol. 4. that can be observed in a more effective manner using spectral IEEE, 2004, pp. 2051–2056. imaging. During the aging process, the chemical properties of [13] T. Taxt and A. Lundervold, “Multispectral analysis of the brain using ink and paper change due to various environmental factors. magnetic resonance imaging,” IEEE Transactions on Medical Imaging, Spectral imaging can potentially capture subtle differences in vol. 13, no. 3, pp. 470–481, 1994. inks or paper due to aging. These are just a few application [14] L. H. Clemmensen, M. E. Hansen, and B. K. Ersbøll, “A comparison of examples where HSI can potentially provide solutions to some dimension reduction methods with application to multi-spectral images major practical problems in document analysis. We hope of sand used in concrete,” Machine Vision and Applications, vol. 21, no. 6, pp. 959–968, 2010. that this work will open up many exciting possibilities for [15] D. G. Zawada, “Image processing of underwater multispectral imagery,” tackling forensic document examination problems with a new IEEE Journal of Oceanic Engineering, vol. 28, no. 4, pp. 583–594, perspective. 2003. [16] H. H. Thodberg, “A review of bayesian neural networks with an application to near infrared spectroscopy,” IEEE Transactions on Neural ACKNOWLEDGMENT Networks, vol. 7, no. 1, pp. 56–72, 1996. [17] M. Alouini, “Target detection and discrimination through active multi- This research was supported by ARC Grant DP110102399. spectral polarimetric imaging,” in Computational Optical Sensing and Imaging. Optical Society of America, 2005, pp. 1–8. [18] S. Joo Kim, F. Deng, and M. S. Brown, “Visual enhancement of old R EFERENCES documents with hyperspectral imaging,” Pattern Recognition, vol. 44, [1] P. R. Martin, “Retinal color vision in primates,” in Encyclopedia of no. 7, pp. 1461–1469, 2011. Neuroscience. Springer, 2009, pp. 3497–3501. [19] G. Edelman, E. Gaston, T. van Leeuwen, P. Cullen, and M. Aalders, [2] P. Shippert, “Introduction to hyperspectral image analysis,” Online “Hyperspectral imaging for non-contact analysis of forensic traces,” Journal of Space Communication, vol. 3, 2003. Forensic Science International, vol. 223, pp. 28–39, 2012. [3] S. Baronti, A. Casini, F. Lotti, and S. Porcinai, “Principal component [20] E. B. Brauns and R. B. Dyer, “Fourier transform hyperspectral visi- analysis of visible and near-infrared multispectral images of works of ble imaging and the nondestructive analysis of potentially fraudulent art,” Chemometrics and Intelligent Laboratory Systems, vol. 39, no. 1, documents,” Applied spectroscopy, vol. 60, no. 8, pp. 833–840, 2006. pp. 103–114, 1997. [21] R. Padoan, T. A. Steemers, M. Klein, B. Aalderink, and G. de Bruin, [4] B. Thai and G. Healey, “Invariant subpixel material detection in “Quantitative hyperspectral imaging of historical documents: technique hyperspectral imagery,” IEEE Transactions on Geoscience and Remote and applications,” ART Proceedings, 2008. Sensing, vol. 40, no. 3, pp. 599–608, 2002. [22] M. E. Klein, B. J. Aalderink, R. Padoan, G. De Bruin, and T. A. Steemers, “Quantitative hyperspectral reflectance imaging,” Sensors, [5] D. Han, Z. Guo, and D. Zhang, “Multispectral palmprint recognition vol. 8, no. 9, pp. 5576–5618, 2008. using wavelet-based image fusion,” in Proc. International Conference on Signal Processing. IEEE, 2008, pp. 2074–2077. [23] foster + freeman, http://www.fosterfreeman.com/index.php. [6] Y. Hao, Z. Sun, T. Tan, and C. Ren, “Multispectral palm image fusion [24] ChemImage, http://www.chemimage.com/. for accurate contact-free palmprint recognition,” in Proc. International [25] D. L. Hammond, “Validation of lab color mode as a nondestructive Conference on Image Processing. IEEE, 2008, pp. 281–284. method to differentiate black ballpoint pen inks*,” Journal of forensic [7] R. K. Rowe, K. Nixon, and S. Corcoran, “Multispectral fingerprint sciences, vol. 52, no. 4, pp. 967–973, 2007. biometrics,” in Proc. IEEE SMC Information Assurance Workshop. [26] Z. Khan, F. Shafait, and A. Mian, “Hyperspectral imaging for ink IEEE, 2005, pp. 14–20. mismatch detection,” in Proc. International Conference on Document [8] Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Face recognition in Analysis and Recognition (ICDAR), 2013. hyperspectral images,” IEEE Trans. on Pattern Analysis and Machine [27] F. Shafait, D. Keysers, and T. M. Breuel, “Efficient implementation of Intelligence, vol. 25, no. 12, pp. 1552–1560, 2003. local adaptive thresholding techniques using integral images,” Document [9] H. Chang, A. Koschan, M. Abidi, S. G. Kong, and C.-H. Won, Recognition and Retrieval XV, pp. 681 510–681 510–6, 2008. “Multispectral visible and infrared imaging for face recognition,” in [28] K. Franke, O. Bunnemeyer, and T. Sy, “Ink texture analysis for writer Proc. Computer Vision and Pattern Recognition Workshops. IEEE, identification,” in Proc. IEEE Workshop on Frontiers in Handwriting 2008, pp. 1–6. Recognition, 2002, pp. 268–273. [10] C. Boyce, A. Ross, M. Monaco, L. Hornak, and X. Li, “Multispectral [29] K. Franke and S. Rose, “Ink-deposition model: The relation of writing iris analysis: A preliminary study,” in Proc. Computer Vision and and ink deposition processes,” in Proc. IEEE Workshop on Frontiers in Pattern Recognition Workshops (CVPRW). IEEE, 2006. Handwriting Recognition, 2004, pp. 173–178.