When printing a digital color image using a printing device, 24-bit RGB data are typically converted to CMYK halftone data by printer driver processing. The converted halftone pattern may change depending on the version of the operating system or printer driver. However, this change may be visually recognized as a difference in the printed image, which is problematic. Hence, a perceptual collation system that detects only differences in visually perceived images is required. Herein, we propose a perceptual collation method to automate the detection of differences in multilevel halftone images based on human assessment. We use the S-CIELAB metric to calculate the color difference by considering human visual characteristics. The proposed perceptual image collation is developed based on the color difference and its gradient features. The experimental results for 25 test image sets show an overall error rate of 8%, thereby verifying the effectiveness of the proposed method.
In image printing, a unique output image must be obtained from the input digital image data. This is particularly critical in the printing industry and must be prioritized to avoid customer complaints. However, this task is extremely difficult to achieve. One of the causes of differences in output occurs when 24-bit RGB image data are converted to CMYK halftone image data by printing processing. Differences in the computer operating system (OS) or printer driver used in this process result in differences in drawing processing and calculation accuracy, thereby resulting in different output halftone patterns.
Figure 1 shows an example of different halftone images converted from the same 24-bit RGB standard test pattern [1]. Figures 1(a) and (b) show the converted halftone images for a 32-bit Windows OS and a 64-bit OS, respectively. For each figure, the left image shows the entire image, and the area indicated by a blue circle is enlarged in the image on the right. In Fig. 1(a), the frame line is filled with only black color, whereas in Fig. 1b, it is represented by a mix of CMYK. In addition to the different color representations, the thickness and position of the border may change. In recent years, owing to the release of new OSs and drivers as well as hardware updates, obtaining a unique output halftone pattern from an input image has become particularly challenging. (a) 32-bit OS (b) 64-bit OS
In the printer manufacturing industry, different output image qualities result in customer complaints. Therefore, when an OS or printer driver is updated, new output images must be compared with previous output images. Moreover, inspection is required every time a customer issues a complaint. Meanwhile, image collation can be easily and automatically obtained from the difference images. Therefore, images with slightly different halftone patterns can be detected. However, it is costly to develop and distribute modified software for all OSs or driver updates. Hence, if different output halftone images cannot be visually recognized, referred to as perceptual collation herein, it indicates that the industry has not modified or distributed the relevant software.
Generally, experts will visually inspect the difference between an output image and a test image and then assess whether the difference is insignificant. However, visual collation requires a significant amount of effort. Hence, an automatic perceptual collation system must be developed to inspect output images rather than human observation. Herein, we propose a perceptual collation method for multilevel color halftone images based on visual characteristics.
The color difference is calculated from two input CMYK halftone images based on human visual characteristics. Subsequently, two types of features are extracted from the color difference image. Finally, a perceptual collation was performed. If a noticeable difference is observed, then a “no good (NG)” response is output. However, if no obvious difference is observed, an “OK” response is output. Figure 2 shows an outline of the proposed method. Human visual characteristics must be considered for automatic human collation. Human color perception exhibits different spatial-frequency characteristics of luminance and opponent color (RG, BY) components [2]. The CIELAB color system, a widely used system, cannot evaluate visual characteristics because its color space does not consider spatial characteristics. Zhang and Wandell [3] proposed a spatial extension to CIELAB (S-CIELAB) to account for the effects of spatial patterns on color appearance and discrimination. The spatial extension preprocesses the input images before applying the standard CIELAB color difference formula. Each luminance and opponent color component is passed through a spatial filter that is selected according to the spatial sensitivity of the human eye for that color component. Subsequently, the final filtered images are transformed to the XYZ format such that the standard CIELAB color difference formula can be applied. S-CIELAB has been applied in various scenarios [4]– [6]. Figure 3 shows an outline for the derivation of S-CIELAB pixel values for a test pattern in [1].
First, a pair of CMYK images are converted to RGB values using the following relation [7]: = 1 − = 1 − = 1 − (1, × (1 − ) + ) (1, × (1 − ) + ) (1, × (1 − ) + ) (1) Subsequently, RGB images are converted to LMS cone responsivities based on the SCIELAB Matlab implementation as follows: = 0.0079 0.0009 each opponent color plane ( = 1, 2, 3).
Once a pair of CMYK images is transformed into the opponent color space, the images are spatially filtered using filters that approximate the contrast sensitivity functions of the human visual system. In this study, we performed this filtering via convolution in the spatial domain. In each opponent component, the filter is a linear combination of weighted exponential functions; its kernel sums to 1 in the form of a Gaussian function series. The following equations show the spatial form of the convolution kernels for In the discrete implementation, the scale factor is selected such that sums to 1. Moreover, the scale factor is selected such that for each opponent’s color plane, its two-dimensional kernel sums to one. The parameters and represent the weight and spread (in degrees of visual angle) of the Gaussian functions, respectively. The tri-stimulus values X, Y, and Z are calculated from these images by inverse transformation and converted into L*, a*, and b*, respectively. The color difference ΔE based on the visual characteristics is calculated as follows
∆ ∗ = {(∆ ∗)2 + (∆ ∗)2 + (∆ ∗)2}2 1 where Xn, Yn, and Zn are the corresponding tri-stimulus values of the perfect reflective surface.
Using the aforementioned procedure, the L*, a*, and b* values in the perceived image space are calculated at each pixel, and the color difference based on the visual characteristics are derived for a pair of halftone images by comparing each pixel. 2.2
We propose a perceptual collation method using the perceptual color difference obtained in the previous section.
In this method, collation is performed at each local block based on visual characteristics. The perceptual collation on a block-by-block basis is inspired by the comparison of local parts rather than by viewing the entire image. In this method, a two-stage collation is performed. In the first stage, the average value of the gradient of the perceptual color difference in a local block is used as a feature value because it is difficult to recognize the difference when the color difference has equal spatial fluctuations. In the second stage, the average value of the absolute perceptual color difference in a local block is used as a feature value because a large color difference is perceived even if the gradient of the color difference is small.
In this study, we considered the block size. The retina of the human eye comprises a region of densely packed cone cells known as the fovea [8]. The fovea corresponds to a circular area at the center of the visual field at 2° and possesses the greatest visual acuity. Therefore, the block size of the image should be smaller than the visual field at 2°. Specifically, the number of pixels N corresponding to the diameter of the center 2° field-of-view is obtained as follows: ( ) = ( ) × tan 180 × ( 2.54 ) ( ℎ) × 2 (6) A square with the number of pixels N as the length of one side is the size that encompasses the field-of-view. In this study, a square with N / 2 as the length of one side is used for the block.
Collation was performed for each block. Specifically, it was performed while moving the target block by raster scanning. If a block is set on a tile, the result may be unstable, depending on the block position. Therefore, in this study, a raster scan was performed while overlapping the blocks by half.
In the first stage, we focused on the gradient of the perceptual color difference. We applied the Prewitt filter to the perceptual color difference of each pixel obtained in the previous section. Two gradient images, Ix (x, y) and Iy (x, y), were detected in the vertical and horizontal directions, respectively. Hence, the total gradient value I(x, y) in both the vertical and horizontal directions is obtained using the following equation: (7) The average value of the gradient ∇fi is calculated for each block i. If a block with ∇fi > T1 exists, then the final result becomes “NG.” Here, T1 is the threshold for the collation.
In the second stage, image collation is performed on the blocks that are “OK” in the first stage, using the absolute color difference. In the first stage, we detected the area where the gradient of the perceptual color difference was large. However, even when the color difference fluctuated equally in the local area, the difference was visible with a large perceptual color difference. When the average color difference Δ of a block i satisfies Δ > 2, the final result becomes “NG.” Here, T2 is the threshold for collation.
In Stages 1 and 2, an image pair for which no “NG” is detected in any block is determined to be “OK.” 3 3.1
In the experiment, 25 standard test images were used. The input images comprised two types of 2-bit CMYK halftone images converted by different OSs or drivers from an identical digital RGB image. The two images have different halftone patterns. In other words, the difference image between the two images was not 0 in each test image. Figure 4 shows the partial test images selected from [1].
In a preliminary experiment, a collation assessment was performed on all the test images by experts, and the results were used as the ground truth. In the main experiment, we verified the experts’ results using the proposed method. The parameters used for SCIELAB spatial filtering were set when a document printed at a resolution of 600 dpi was viewed at a viewing distance of 50 cm. Meanwhile, the block size used for collation was 206 × 206 pixels. Because the matching costs were asymmetric, each threshold parameter value was determined under a zero false negative. 3.2
Figure 5 shows an example of the collation result for Sample 10. Sample 10 is an image of a natural scene with a red flower at the center. Figures 5(a) and (b) show the pair of input images, and Figs. 5(c) and (d) show their close-up images, where the color difference is particularly large. Figures 5(e) and (f) represent the gradient and color difference feature values in the first and second stages, respectively, as grayscale images. “NG” blocks are represented in red. Color differences exist between Figs. 5 (e) and (f). However, the difference is not visible in Figs. 5(c) and (d). Ground truth was “OK,” and the difference between these images was acceptable. In our collation results in Figs. 5(e) and (f), no red blocks were detected, and the final result was “OK.” (a) Sample A (b) Sample B
Figure 6 shows an example of the collation result for Sample 24. The image was not natural, but a poster image was drawn for illustration. Comparing Figs. 6(c) and (d), a slight difference was observed in the color depth below the center as Fig. 6(d) is darker than Fig. 6(c). Meanwhile, the ground truth was “NG,” and the difference between these images was unacceptable. In our collation results shown in Figs. 6 (e) and (f), red blocks were detected, and the final result was “NG.” Hence, the proposed method was effective not only for natural images but also for poster images.
(a) Sample A (b) Sample B (a) Sample A (b) Sample B (e) First stage (Gradient) (f) Second stage (Color difference)
Fig. 7. Results for Sample 9. (a) Sample A (b) Sample B (c) First stage (Gradient) (d) Second stage (Color difference)
Fig. 8. Results for Sample 15. (a) Close-up image of Fig. 8(a)
(b) Close-up image of Fig. 8(b) (c) Close-up image of Fig. 8(a) (d) Close-up image of Fig. 8(b)
Fig. 9. Close-up images around misdetected areas. 4
Herein, we proposed a perceptual image collation method that detects human-visible differences caused by OSs and drivers used when converting RGB digital images into CMYK multilevel halftone data. Using the S-CIELAB measure, the color difference was calculated based on human visual characteristics. Image collation was performed in two stages: color difference gradient collation and color difference collation. We verified our method using 25 pairs of test images converted using different OSs and drivers. The error rate was 14.3% for false positives under a false negative rate of 0%, and the total error rate was 8%, thereby confirming the effectiveness of the proposed method. Furthermore, we verified the robustness of the proposed method by applying it to large image data.
The proposed method only performs judgments on digital image data. Therefore, it does not consider the color reproduction and noise caused by the actual printing process. For a practical system construction, these factors must be considered in future studies.