Testing a Banknote Checking System Bernhard Blaschitz and Daniel Soukup and Harald Penz and Werner Krattenthaler and Reinhold Huber-Mörk 1 Abstract. We present our work in progress in the direction of gener- ating realistic, challenging and diverse test data in order to evaluate a banknote checking system. Test image generation can be divided into, firstly, the design and composition of highly diverse patterns to Generate form the digital design of a banknote, secondly, the application of different variations which are caused by ink, paper and physically print patterns printing the note and, thirdly, the simulation of the subsequent image acquisition, which includes variations due to banknote transport, illu- mination, camera optics and electronics. These simulations are based on and compared to scans of demo banknotes. We present initial re- sults in simulation of banknote paper, printing and image acquisition. We demonstrate the approach in an initial investigation on synthetic defect generation and its impact on banknote checking results. Al- Paper, gorithmic validation of checking routines is also demonstrated. First Printing, Simulate paper conclusions are drawn and further work is discussed. Printing and printing Defects 1 Introduction High quality standards in banknote production are commonly achieved by optical inspection systems. As large numbers of ban- Transport, knotes are printed each year, e.g. 8 billion notes in Eurosystem in Illumi- Simulate image 2014, the quality inspection system has to run at high speed. Please note that counterfeit detection is not an issue for our work. The va- nation, acquisition riety of printing processes, e.g. intaglio and offset printing [10], and Camera security features complicates the inspection system. Additional vari- Optics ations are introduced by different printing presses as well as vari- ation in banknote paper etc. Finally, the image acquisition process itself also introduces variations. The common approach in optical banknote inspection is to define a set of so called fit banknotes to be accepted by the banknote inspection system, i.e. the system is trained on these. Any significant deviation from this set renders a banknote unfit [6]. In this paper we present our work in progress in the direction of Banknote generating realistic, challenging and diverse test data [1] in order to checking system evaluate a banknote checking system. For this, a software framework has been set up with two main goals in mind. Firstly, to evaluate the results of banknote checking systems when certain defects are applied to its input images and, secondly, to experiment with and improve on existing algorithms. Fig. 1 shows the components of the presented test system for banknote checking. This paper is organized as follows. Sec. 2 introduces our modular Figure 1. Subsystems of our test system for banknote checking: test pattern generation, simulation of the production of banknotes, simulation of approach to generate test data. Some use cases of our tool are dis- the image acquisition, which is usually done with a line scan camera that cussed in Sec. 3, and Sec. 4 summarizes the state of the work and includes a transport mechanism. The original software that makes up the suggests future work. banknote checking system itself is used and evaluated. 1 Digital Safety & Security Department, AIT Austrian Institute of Technol- ogy GmbH, Donau-City-Straße 1, 1220 Vienna, Austria Figure 3. A sample of demo banknotes provided by Oesterreichische Banknoten und Sicherheitsdruckerei (OeBS) that were used for scanning in order to have meaningful statistics about print variations. Figure 2. An artificial banknote pattern composed of 32 different tiles, which mimic high-resolution features of intaglio printing. 2.2 Simulating Banknote Paper and Printing 2 Test Data Generation To derive a realistic model of all the different factors that cause prints to look differently, we took a reverse engineering approach The central aspect of this work is to generate realistic, challenging and scanned the front and back side of 110 demo banknotes at a res- and diverse test data that is then entered into the banknote checking olution of 1016 dpi with a standard Epson scanner, see Fig. 3. system. We then cut out blank banknote regions; the average pixel value in a blank region of a banknote (”allonge”) allows for calibrating the Test image generation can be divided into color values. After subtracting this average, the remaining deviations show how inhomogeneous white paper is. 1. the design and composition of patterns to form the digital design The material shows a certain ”cloudiness” that can be seen when the of a banknote with a high variability, which will be presented in contrast in the gray value image is enlarged (see Fig. 4, top). This Sec. 2.1, behavior can be quantified using a set of co-occurrence matrices [8]. 2. application of different variations which are caused by physically From the 110 scanned images, which show quite some variety, we printing the note, which usually happens in different phases and generate an even larger number by taking convex combinations of includes variabilities due to paper, ink and the press itself, see these and then use the texture synthesis algorithm of [3] to generate Sec. 2.2 and new synthetic paper images (see Fig. 4, bottom). 3. image acquisition, which includes transport, (varying) illumina- We plan to do the same with printed regions of the banknote scans tion conditions and camera optics and electronics, as well as the in order to receive a model for color variations. sensing itself, see Sec. 2.3. The banknote checking system receives digital images of a ban- 2.3 Simulating Image Acquisition knote as an input, performs several tests and returns a fit/unfit deci- There is a number of effects that have to be considered when mod- sion. With the aid of this work, we can evaluate and improve existing eling the image acquisition process [4, 5]. Besides of optical effects algorithms, see Sec. 3. e.g. focus, lens distortions, and lens aberrations, there are further ef- fects caused by the camera chip like camera-noise, dynamic range, 2.1 Pattern Generation quantization, integration over the pixel area and, as the case may be, demosaicing. Also illumination variations – e.g. caused by aging of In order to have test images, whose print and acquisition could be illumination units – influence the appearance of images. Moreover, simulated, a set of exemplary patterns was developed. Our print pat- as banknotes are often acquired in a line-scan inspection process, terns are subdivided into tiles. The reference image in Fig. 2 mimic distortions caused by the transport play a crucial role. especially high-resolution features of intaglio printing [9], e.g. Guil- We started modeling the image sharpness given a certain camera loche patterns. Some tiles with extreme artificial test patterns are also and optics. The image sharpness results from the lens’ sharpness included. combined with the pixel resolution of the camera chip and possi- The considered resolution is 0.2 mm/pixel (127 dpi), which is typ- ble transport blur. Measuring the sharpness of an image is done by ical for current banknote checking systems [7]. However, the ref- means of the slanted edge method [2] for digital camera and scanner erence images (see Fig. 2) are designed in eightfold precision, that analysis. The slanted edges are sharp black/white edges on a calibra- is 0.025 mm/pixel (1016 dpi), which allows for introducing errors in tion chart (Fig. 5). In the camera image that edge appears blurred due sub-pixel accuracy and simulating aliasing and other digitizing prob- to the mentioned effects. By measuring the blur of the edge within lems. The high resolution images are rotated in the range from −5◦ the image perpendicular to the edge itself, one can derive the spa- to +5◦ with an interval of 0.05◦ and (separately) shifted by 0 to 15 tial resolution, i.e. sharpness, of the camera system. A robust edge pixels in x- and y-direction with an interval of 1 pixel (0.025 mm). spread function (ESF) measurement can be obtained by combining a The reason for using the range of 0 to 15 pixels for shifting is that number of measurements along an edge. this covers the range of 2 pixel in the considered resolution. These 2 The calibration sheet image, artificially generated in high- pixels are advisable because of a following down-sampling step. resolution, is blurred with a number of different Gaussian kernels. Each blurred image is down-sampled to target resolution and the result’s sharpness is measured in horizontal and vertical directions. Those measures are compared with the measured sharpness values from the camera system’s image. The chosen Gauss-Kernel is the one resulting in the smallest sharpness difference. By means of that identified blurring kernel it is possible to simulate the sharpness of the target camera system for arbitrary patterns, artificially generated or scanned ones. Additionally we modeled the camera noise, as this effect can be relatively easily measured for the real camera system. A noise pattern comprising the same statistical data was then be generated and added to modeled target images. Figure 5. Our calibration sheet used for slanted edge sharpness measurements comprising slanted sharp edges in (near) horizontal and vertical directions, respectively. 3 Testing the Banknote Checking System With the synthetic images of Sec. 2, we carried out experiments for the two main goals of our testing software, namely evaluate banknote 3.2 Improving Existing Algorithms checking systems and experiment with inspection algorithms. The camera resolution is 0.2 mm/pixel, but in our inspection system the pixel by pixel inspection is done in 0.4 mm/pixel. The images are 3.1 Evaluating the System with Synthetic Data warped (rectification to compensate shifts and rotations) and reduced to this resolution. The reduction is done by applying a smoothing We placed 8 ∗ 16 = 128 exemplary defects in form of red dots on our filter and selecting every second pixel (we call this shrinking). We synthetic banknote, see Fig. 6, top. This was done at high resolution, have therefore the steps warp, smooth and shrink. It is inevitable to where each dot has a diameter of 48 pixels (1.2 mm). The red dots in smooth before shrinking but it is not that clear when warping should the top row have sharp edges. The sharpness decreases from top to be done. It has been suspected that it is best to warp first. We verified bottom. The blending factor decreases for each column from α = 1 this with our test system. on the left to α = 1/16 on the right. The currently checking system is heavily adopted and optimized to The banknote checking system was trained using 456 synthetic specialized hardware. We have therefore reimplemented some parts generated banknotes (256 translations and 200 rotations) with added of the inspection process in our simulation. This code is less effi- random noise. The banknote with the added red dot defects was then cient but much more flexible. It was then easy to test different orders checked resulting in Fig. 6, bottom. The red and green markings show of warp, smooth and shrink. The results are presented in Fig. 7. It found defects. Each severe defects (red marking) alone would be shows pixel by pixel the gaps between lowest and highest intensity sufficient to classify the banknote as unfit. Moderate defects (green values in the training set. Large differences (dark pixels in Fig. 7) in- marking) are acceptable if there number is not too big. dicate wide ranges of allowed pixel values which means less sensible inspection. It can be observed that is best to warp first (Fig. 7 (e) and (f)). But it is acceptable to warp between smoothing and shrinking. This may not look very appropriate but can help to optimize run- time when warping (pixel interpolation) is computational expensive with the used hardware. The gray backgrounds in the images on the right side of Fig. 7 are caused by the artificial noise added to the test images. 4 Conclusions & Future Work As most of this paper is work-in-progress, only preliminary results in test data validation could be presented here. However, it could be shown that banknote checking systems can be benchmarked with simulated printing defects using synthetic data. Furthermore, an evaluation for the order in which specific steps of the checking system should be applied, was carried out. Aside from implementing missing steps that were already de- scribed in the previous sections, we plan 1. to carry out the whole testing with synthetic color images and to inspect the color of the print itself, 2. to use more realistic printing defects like blank spots/streaks, Figure 4. ”Cloudiness” of original banknote paper (top, amplified blade streaks or hickeys and contrast) and synthetic (bottom, amplified contrast) texture obtained using [3]. These variations were applied to our artificial banknote patterns in order 3. to investigate relative shifts of the printing plates, i.e. incorporate to simulate the printing process. different printing phases and the alignment of the printing plate w.r.t. the paper sheet. Figure 6. Evaluation results: top: Artificial defects (red dots) with decreasing intensity (left to right) and edge sharpness (top to bottom) on a synthetic banknote. bottom: Defects recognized by our banknote checking system. Severe defects are marked in red and moderate defects in green. The most severe defect is marked by a cyan border (first defect column, defect row 6). (a) kimagek = 34.5 (b) kimagek = 62.5 (c) kimagek = 12.5 (d) kimagek = 44.0 (e) kimagek = 9.5 (f) kimagek = 43.3 Figure 7. Performance evaluation for different orders of processing: Difference images of trained Min- and Max-images are shown. The images are doubled in contrast and inverted (white means 0) for presentation purposes. Left (a c e): without noise. Right (b d f): with added noise. (a) and (b) Smooth → Shrink → Warp. (c) and (d) Smooth → Warp → Shrink. (e) and (f) Warp → Smooth → Shrink. ACKNOWLEDGEMENTS We thank Oesterreichische Banknoten und Sicherheitsdruckerei (OeBS) for providing the demo banknotes. REFERENCES [1] Kevin Bowyer and P Jonathon Phillips, Empirical evaluation tech- niques in computer vision, IEEE Computer Society Press, 1998. [2] Peter D. Burns, ‘Slanted-edge MTF for digital camera and scanner anal- ysis.’, in Proc. of PICS, pp. 135–138. IS&T, (2000). [3] Alexei A Efros and Thomas K Leung, ‘Texture synthesis by non- parametric sampling’, in Proc. ICCV, volume 2, pp. 1033–1038. IEEE, (1999). [4] EMVA. Standard 1288, standard for characterization of image sensors and cameras, 2010. [5] Robert D Fiete, Modeling the imaging chain of digital cameras, SPIE, Bellingham, 2010. [6] Jan-Mark Geusebroek, Peter Markus, and Peter Balke, ‘Learning ban- knote fitness for sorting’, in Proc. of International Conference on Pat- tern Analysis and Intelligent Robotics (ICPAIR), volume 1, pp. 41–46. IEEE, (2011). [7] Giesecke & Devrient, www.gi-de.com/gd_media/media/ en/documents/brochures/processing_1/OBIS-3.pdf, OBIS 3 - High-resolution Optical Banknote Inspection System for BPS R 2000 OBIS. [8] Robert M Haralick, Karthikeyan Shanmugam, et al., ‘Textural features for image classification’, IEEE Transactions on systems, man, and cy- bernetics, (6), 610–621, (1973). [9] J. Hofmann, E. Gillich, H. Dörksen, D. Chassot, Th. Türke, and V. Lo- hweg, ‘New strategies in image processing for standardized intaglio quality analysis in the printing process’, in Proc. ODS, (2014). [10] R. L. van Renesse, Optical document security, Artech House, Boston/London, 3rd edn., 2005.