Manipulation of anisotropic reflections based on optical models using multiple projectors Shogo Ohsumi 1, Toshiyuki Amano 2 1,2 Wakayama University, 930, Sakaedani, Wakayama-shi, Wakayama, Japan Abstract This paper proposes a novel appearance-manipulation technique that parametrically manipu- lates the visible anisotropic reflection property with illumination projection from multiple pro- jectors. This method obtains a reflectance matrix corresponding to the bidirectional reflectance distribution function (BRDF) from images captured using multiple cameras. The reflectance matrix was then fitted to the Ashikhmin BRDF model to estimate its parameters of the BRDF model. The reflectance matrix corresponding to the target BRDF was then calculated by ma- nipulating the estimated parameters. The anisotropic reflection was manipulated based on the optical model by projecting images from multiple projectors that changed the texture repre- sented by the reflectance matrix calculated in this manner. Keywords 1 Anisotropy, Light-field projection, BRDF. 1. Introduction Horizontally aligned projectors and a screen composed of lenticular lenses with a diffusing screen can achieve a projection-based autostereo- The angular light intensity distribution on the scopic display [5,6]. Jones et al. [7] demonstrated surface is formed by its properties (e.g., bidirec- a wide-viewing and high-angular-resolution auto- tional reflectance distribution function (BRDF), stereoscopic 3D display using 216 projectors. Na- bidirectional transmittance distribution function gano et al. [8] proposed an autostereoscopic pro- (BTDF)) and represents rich materiality, such as jection display with 72 overlay images projected glossy metallic reflection, clear glass caustic, and onto a vertically oriented lenticular screen with beautiful structural color. Meanwhile, precisely black back. Such front-projection auto-stereo- designed light-field projection, instead of normal scopic displays can be used to show complex ma- environmental illumination, has the potential to teriality on an object using retroreflection paint on manipulate light angular distribution and alter our a 3D object [9]. However, they only displayed a perception of materiality [1-4]. Such material ap- BRDF and did not realize the alternation or ma- pearance manipulation is a key challenge in spa- nipulation of the BRDF that the object originally tial augmented reality (SAR), known as projection had. mapping. This paper proposes an anisotropic re- flection property manipulation, which is an angu- lar distribution manipulation of the reflected light 2.2. VDDAM ray on an anisotropic reflection surface using light field projection as a novel SAR technique. Amano et al. [10] demonstrated view-direction dependent appearance manipulation (VDDAM), 2. Related work using multiple projector-camera feedback sys- tems. Murakami et al. [11] proposed another 2.1. Auto-stereoscopic display method for VDDAM based on reflectance meas- urement among multiple projectors and cameras, ――――――――――――― which was equivalent to roughly sampled BRDF. 1 Corresponding author. APMAR’22: The 14th Asia-PacificWorkshop on Mixed and Aug- Amano and Yoshioka [12] combined reflectance mented Reality, Dec. 02-03, 2022, Yokohama, Japan analysis with multiple projector-camera feedback EMAIL: s226042@wakayama-u.ac.jp (Shogo Ohsumi); and expanded the applicable reflection property to amano@wakayama-u.ac.jp (Toshiyuki Amano) ORCID: 0000-0002-1393-8757 (Shogo Ohsumi); 0000-0003- retroreflection and improved robustness against 4146-1375 (Toshiyuki Amano) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ht tp: // ceur -ws .or g Works hop I SSN1613- 0073 Pr oceedi ngs environmental lighting changes. However, the pa- rameters for anisotropy are unknown for Amano et al. [10], whereas Murakami et al. [11] and Amano and Yoshioka [12] must consider in ad- vance what type of anisotropy was used to create the target image. In this paper, we propose a method to parame- terize the VDDAM applying the Ashikhmin BRDF model to the previously acquired reflection characteristics. With this parameterization, we en- hanced or reduced the anisotropy and then recre- ated the reflectance. Finally, we calculated the manipulation references for each viewing direc- tion using the recreated reflectance. Figure 1: Experimental devices. 2.3. Reflectance matrix In this section, we introduce the Ashikhmin BRDF model to fit the reflectance matrix. The Murakami and Amano proposed a response Ashikhmin BRDF model is described by the sum model for multiple projectors and cameras that of the specular and diffuse components, with the considers color [11]. The RGB values at a point A specular component 𝜌= defined as follows: in the captured and projected images are defined 𝜌! (𝐤" , 𝐤 # ) ('!(𝐡𝐮)"+'#(𝐡𝐯)" ) as follows: = ((𝑛$ + 1)(𝑛% + 1) (𝐧𝐡) ("-(𝐡𝐧)" ) / 5𝑅 + (1 − 𝑅! ) 31 − (𝐤𝐡)4 8 , (7) 8𝜋 (𝐡𝐤) 𝑚𝑎𝑥3(𝐧𝐤" ), (𝐧𝐤 # )4 ! # % # 𝐂"! = $𝑐!" , 𝑐! , 𝑐!$ ' , 𝑐!" ≥ 0, 𝑐! ≥ 0, 𝑐!$ ≥ 0, where 𝐤> and 𝐤 ? represent normalized vector to 𝑤ℎ𝑒𝑟𝑒 𝑖 = 1,2, … , 𝑢, (1) the light and viewer. 𝐡 represent normalized half- % vector between 𝐤> and 𝐤 ? . 𝐧 represent surface 7& = 8𝑝&" , 𝑝&# , 𝑝&$ : 𝐏 # , 𝑝&" ≥ 0, 𝑝& ≥ 0, 𝑝&$ ≥ 0, normal to macroscopic surface. 𝑛@ and 𝑛A repre- 𝑤ℎ𝑒𝑟𝑒 𝑗 = 1,2, … , 𝑣, (2) sent two phong-like exponents that control the where 𝑢 denotes the number of cameras, and 𝑣 specular lobe shape. The larger the value of 𝑛@ , denotes the number of projectors. In this case, by the higher the directivity of reflection in the u di- expressing the reflection at an object surface as a rection. Similarly, the larger the value of 𝑛A , the 7 ∈ ℛ'×' , it can be described as follows: matrix 𝐾 higher the directivity of reflection in the v direc- tion. 𝐂"! = 𝐾 7!& 𝑀!& 𝐏 7& , (3) where the matrix 𝑀!& represents the color-mixing 3. Proposed method matrix [13] that calibrates the color. Furthermore, when multiple projectors or cameras are used, Our proposed method obtained a reflectance they are represented as follows: matrix representing the optical response between projectors and cameras corresponding to a 𝐂B = 𝐾 C𝑀C𝐏 C, (4) roughly sampled BRDF on every single point on where the object's surface with the experimental devices " shown in Figure 1. Subsequently, we fitted the re- 𝐂" = $𝐂%!" , 𝐂%#" , … , 𝐂%$" ( , 𝐏 + = $𝐏 ,!" , 𝐏 ,$" (" , ,#" , … , 𝐏 (5) flectance matrix with the Ashikhmin BRDF 𝑀!! 𝑀!" ⋯ 𝑀!# 6!! 𝐾 6!" 𝐾 ⋯ 6!# 𝐾 model and parameterized the reflectance relation- * = ,𝑀"! 𝑀"" ⋯ 𝑀"# * ⎛ 6"! 𝐾 6"" 𝐾 ⋯ 6"# ⎞ 𝐾 ship. Then, the parameters were manipulated to 𝑀 0 ,𝐾 = . (6) ⋮ ⋮ ⋱ ⋮ ⋮ ⋮ ⋱ ⋮ 𝑀$! 𝑀$" ⋯ 𝑀$# ⎝𝐾6$! 6$" 𝐾 ⋯ 6$# ⎠ 𝐾 design a desired anisotropic reflection, yielding a recreated reflection matrix. Finally, the VDDAM Hereafter, we regard the color spaces as calibrated, based on the optical model achieved the desired and we write 𝑀C𝐏C as 𝐏 C in the following sections. appearance that the reflectance matrix represented by projecting images from multiple projectors. 2.4. Ashikhmin BRDF model[14] 3.1. Multiple projector-camera sys- tems In this paper, we employed 7 cameras (Ximea， MQ013CGE2, resolution: 1280 × 1024 ) and 7 projectors (EPSON,EB-W05, resolution:1280 × 800) in order to achieve high quality perceptual BRDF manipulation with complex reflection characteristics. The cameras and projectors were placed in front of the target object and the other projectors are placed radially at 15𝑑𝑒𝑔. intervals around the projector 4 (Figure 1). The cameras are Figure 2: Operating object and an estimate of the 𝑛: placed close to each projector. This arrangement and 𝑛< distributions. takes into account the measurement and manipu- lation of anisotropic reflections. where 𝑠 ∈ (𝑟, 𝑔, 𝑏), 𝑡 ∈ (𝑟, 𝑔, 𝑏), and 𝑘&'"# repre- To obtain the reflectance matrix, first capture sents the reflectance of the 𝑠 color component of the red projection from one projector with all cam- the 𝑡 color projection of projector 𝑗 captured by eras. Similarly, the green and blue projection from camera 𝑖. 𝑛@=B , 𝑛A=B represent anisotropic scattering one projector is captured by all cameras. This pro- for each direction, and u, v, 𝑅C=B , 𝑅==B represent the cess is repeated with seven projectors. The reflec- intensities of the diffuse and specular components, tance is obtained by dividing the image thus ac- respectively. Because the four parameters 𝑛@ , 𝑛A , quired by the RGB of the projection image. 𝑅= , and 𝑅C must be positive, we applied a non- negative condition to the Levenberg-Marquardt 3.2. Manipulation Target Object method. This allows us to obtain the four parame- ters of the Ashikhmin model for a single pixel from the reflectance matrix at a single pixel. We used a drawing foil of Nishijin silk textile, Figure 2 shows the estimated 𝑛@ and 𝑛A . Be- which contains patterns of birds, flowers, clouds, cause there is no significant difference the color and a mountain with rivers, as the manipulation channel, this figure shows only the R channel. target. Various threads, including gold thread and Brightness expresses the value of each parameter, dyed thread, are used in this Nishijin silk textile, and the brighter area has a sharp specular reflec- and differences in gloss can be seen, such as the tion along each direction. A small difference be- gold thread being more reflective than the dyed tween the 𝑛@ and 𝑛A values indicates isotropy, thread. The weaving method also causes differ- whereas a large difference indicates anisotropy. ences in reflectance characteristics. In the case of Area (a) in the figure shows a twill weave using twill weave, the ratio of warp to weft threads on gold threads and has almost isotropic reflections. the surface is close, resulting in isotropic reflec- Area (b) is a satin weave that uses gold threads tions. On the other hand, a satin weave has a and exhibits strong anisotropy. Area (c) is a satin higher ratio of warp threads than weft threads on weave using dyed threads, and it has weak anisot- the surface, resulting in anisotropic reflections. In ropy. Area (d) is a twill weave using dyed threads this study, we regard the target object as a plane. and exhibits diffuse reflection. 3.3. Parameter estimation 3.4. Anisotropy manipulation We employed the Levenberg-Marquardt Our anisotropic manipulation aims to enhance method, a nonlinear optimization scheme, and ob- or reduce its reflection of the anisotropic reflec- tained anisotropic reflection parameters by mini- tion optically while maintaining the glossiness of mizing the error function as follows: the isotropic reflection. Based on this, we updated 𝐸(𝑛!"# , 𝑛$"# , 𝑅%"# , 𝑅""# ) the parameters as follows: , * "# 𝑛; = 𝑛: + 𝛼(𝑛: − 𝑛< ), 𝑛<; = 𝑛< (𝑛: ≥ 𝑛< ) = ) )(𝑘&' − 𝜌(𝐤(' , 𝐤 )& ; 𝑛!"# , 𝑛$"# , 𝑅%"# , 𝑅""# ))) , (8) ! : . (9) ; &+( '+( 𝑛: = 𝑛: , 𝑛<; = 𝑛< + 𝛼(𝑛< − 𝑛: ) (𝑛: < 𝑛< ) (a) Anisotropic reduction (b) Anisotropic enhancement (a) Anisotropic reduction (b) Anisotropic enhancement Figure 3:Target images. Figure 4: Projection images. (a) Anisotropic reduction (b) Original appearance (c) Anisotropic enhancement Figure 5: Projection results. If 𝑛@ was greater than 𝑛A , the difference, which is an anisotropy, was added to 𝑛@ with a multipli- cation of the scaling factor α. Otherwise, the dif- ference was added to 𝑛A as well. When 𝛼 > 1, the difference was expanded, and anisotropy was en- hanced. On the contrary, the anisotropy was com- pletely removed when 𝛼 = −1. We applied this manipulation to the estimated parameters and obtained the desired target images for the entire manipulation area. 3.5. Calculation of projection image (a) Satin weave (b) Twill weave Figure 6: Variation of brightness for each viewpoint. From the reflectance matrix and the target im- ages 𝐂BB = (𝐂BB> % B% , 𝐂B? , … , 𝐂BB@ % % ) , we obtained the projection images 𝐏 CB = (𝐏 CB> % C% CB@ , 𝐏B? , … , 𝐏 % % ) for 4. Result each projector. However, the projection images CB must be positive. Therefore, in this paper, a 𝐏 The target images were created by updating 𝑛@ non-negative conditional optimization problem and 𝑛A , and by manipulating the value of 𝛼 in Eq. min ∥ 𝐾C𝐏CB − 𝐂BB ∥? , (9), the projection images were obtained using Eq. D (10), and then projected. " # $ $ 𝑤ℎ𝑒𝑟𝑒 𝑝B> ≥ 0, 𝑝B> ≥ 0, 𝑝B> ≥ 0, … , 𝑝BA ≥ 0. (10) Figures 3 and 4 shows examples of the target and projection images, respectively. In Figure 5, used the Lawson-Hanson algorithm [15] to obtain the projection results are arranged to correspond the projection value of each projector at a certain to each camera position. All of these images are point. This calculation was performed for all the shown in identical aspects by geometrical trans- points in the operating range to obtain the projec- formation to the common coordinate (cam 4). tion image for each projector. 4.1. Anisotropic enhancement even with anisotropic reduction in the region of the gold thread twill weave. Figure 3 shows the target images created using 𝛼 = 8. We solved the non-negative optimization 5. Conclusion problem described in Section 3.3 and obtained the projection images shown in Figure 4(b). The ma- In this paper, we propose a method for estimat- nipulation results from the projection are shown ing the parameters of the Ashikhmin model from in Figure 5(c). It should be noted that a significant the reflectance matrix. Furthermore, we propose a difference in glossiness between viewing direc- parameter manipulation method that can enhance tions along u and v was observed. Figure 6 shows and reduce anisotropy. Projection images were the brightness change from cam1 to cam5 and calculated using non-negative conditional optimi- cam3 to cam7 when the viewpoint is moved from zation. The projection results showed that anisot- left to right. Figure 6(a) shows the average bright- ropy could be enhanced and reduced in areas with ness of the 3 × 3 pixels in the area shown in Fig- anisotropic reflections. In areas with isotropic re- ure 2(b). When the viewpoint is moved horizon- flections, although the brightness was slightly re- tally in the range of cam1 to cam5 and cam3 to duced, the isotropic reflections were maintained. cam7, the gloss change of the anisotropy en- hanced image is sharper than that of the original References appearance. These results confirmed the enhanced anisotropy. Figure 6(b) shows the average brightness val- [1] S. Shimazu et al., “3d high dynamic range ues of the 3 × 3 pixels in the area shown in Figure display system”, ISMAR, pp.235–236, 2(a). When the viewpoint is moved horizontally (2011). in the range of cam1 to cam5 and cam3 to cam7, [2] T. Amano, H. Kato, “Appearance control us- it can be confirmed that the anisotropic enhance- ing projection with model predictive control”, ment follows the brightness change of the original ICPR, pp.2832–2835, (2010). appearance, although the brightness of the aniso- [3] T. Okazaki et al., “A projector-camera sys- tropic enhancement is reduced compared to the tem for high-quality synthesis of virtual re- original appearance. This confirms that the iso- flectance on real object surfaces”, IPSJ, tropic reflection was maintained in the region of pp.71-83, (2010). the gold thread twill weave. [4] A. J. Law et al., “Perceptually based appear- ance modification for compliant appearance editing”, CG Forum, 30(8), pp.2288-2300, 4.2. Anisotropic reduction (2011). [5] W. Matusik, H. Pfister, “3D TV: a scalable Figure 3(a) shows the target images created system for real-time acquisition, transmis- with 𝛼 = −0.5. The projection images and ma- sion, and autostereoscopic display of dy- nipulation results are shown in Figures 4(a) and namic scenes”, SIGGRAPH, 23(3), pp.814- 5(a), respectively. The overall images are dark- 824, (2004). ened, however the central river area has lost its [6] R. Yang et al., “Toward the light field dis- gloss. As shown in Figure 6(a), when the view- play: Autostereoscopic rendering via a clus- point is moved horizontally in the range of cam1 ter of projectors”, IEEE TVCG, 14(1), to cam5 and cam3 to cam7 in the region shown in pp.84–96, (2008). Figure 2(b), the anisotropic reduction shows a [7] A. Jones et al., “An automultiscopic projec- smaller brightness change than the original ap- tor array for interactive digital humans” SIG- pearance. From these results, we can confirm the GRAPH, (2015). reduction in anisotropy. [8] K. Nagano et al., “An autostereoscopic pro- As shown in Figure 6(b), when the viewpoint jector array optimized for 3D facial display”, is moved horizontally in the range of cam1 to SIGGRAPH, (2013). cam5 and cam3 to cam7 in the region shown in [9] T. Amano, K. Minami, “Structural color dis- Figure 2(a), the anisotropy reduction follows the play on retroreflective objects”, ICAT - brightness change of the original appearance, alt- EGVE, pp. 37– 44, (2015). hough the brightness of the anisotropy decreases compared to the original appearance. This con- firms that isotropic reflection was maintained [10] T. Amano et al., “Viewpoint-Dependent Ap- pearance-Manipulation with Multiple Pro- jector-Camera Systems", ICAT-EGVE, pp.101-107, (2017). [11] K. Murakami, T. Amano, “Materiality Ma- nipulation by Light-Field Projection from Reflectance Analysis”, ICAT-EGVE, pp.99- 105, (2018). [12] T. Amano, H. Yoshioka, “Viewing-Direction Dependent Appearance Manipulation Based on Light-Field Feedback”, EuroVR, pp.192- 205, (2020). [13] S. K. Nayar et al. “A projection system with radiometric compensation for screen imper- fections”, PROCAMS, (2003). [14] M. Ashikhmin, P. Shirley, “An anisotropic phong BRDF model”, Journal of Graphics Tools, (2000). [15] C. L. Lawson, R. J. Hanson, “Solving least squares problems”, (1974).