=Paper=
{{Paper
|id=Vol-3297/paper1
|storemode=property
|title=Manipulation of Anisotropic Reflections Based on Optical Models Using Multiple Projectors
|pdfUrl=https://ceur-ws.org/Vol-3297/paper1.pdf
|volume=Vol-3297
|authors=Shogo Ohsumi,Toshiyuki Amano
|dblpUrl=https://dblp.org/rec/conf/apmar/OhsumiA22
}}
==Manipulation of Anisotropic Reflections Based on Optical Models Using Multiple Projectors==
https://ceur-ws.org/Vol-3297/paper1.pdf
Manipulation of anisotropic reflections based on optical models
using multiple projectors
Shogo Ohsumi 1, Toshiyuki Amano 2
1,2
Wakayama University, 930, Sakaedani, Wakayama-shi, Wakayama, Japan
Abstract
This paper proposes a novel appearance-manipulation technique that parametrically manipu-
lates the visible anisotropic reflection property with illumination projection from multiple pro-
jectors. This method obtains a reflectance matrix corresponding to the bidirectional reflectance
distribution function (BRDF) from images captured using multiple cameras. The reflectance
matrix was then fitted to the Ashikhmin BRDF model to estimate its parameters of the BRDF
model. The reflectance matrix corresponding to the target BRDF was then calculated by ma-
nipulating the estimated parameters. The anisotropic reflection was manipulated based on the
optical model by projecting images from multiple projectors that changed the texture repre-
sented by the reflectance matrix calculated in this manner.
Keywords 1
Anisotropy, Light-field projection, BRDF.
1. Introduction Horizontally aligned projectors and a screen
composed of lenticular lenses with a diffusing
screen can achieve a projection-based autostereo-
The angular light intensity distribution on the
scopic display [5,6]. Jones et al. [7] demonstrated
surface is formed by its properties (e.g., bidirec-
a wide-viewing and high-angular-resolution auto-
tional reflectance distribution function (BRDF),
stereoscopic 3D display using 216 projectors. Na-
bidirectional transmittance distribution function
gano et al. [8] proposed an autostereoscopic pro-
(BTDF)) and represents rich materiality, such as
jection display with 72 overlay images projected
glossy metallic reflection, clear glass caustic, and
onto a vertically oriented lenticular screen with
beautiful structural color. Meanwhile, precisely
black back. Such front-projection auto-stereo-
designed light-field projection, instead of normal
scopic displays can be used to show complex ma-
environmental illumination, has the potential to
teriality on an object using retroreflection paint on
manipulate light angular distribution and alter our
a 3D object [9]. However, they only displayed a
perception of materiality [1-4]. Such material ap-
BRDF and did not realize the alternation or ma-
pearance manipulation is a key challenge in spa-
nipulation of the BRDF that the object originally
tial augmented reality (SAR), known as projection
had.
mapping. This paper proposes an anisotropic re-
flection property manipulation, which is an angu-
lar distribution manipulation of the reflected light 2.2. VDDAM
ray on an anisotropic reflection surface using light
field projection as a novel SAR technique. Amano et al. [10] demonstrated view-direction
dependent appearance manipulation (VDDAM),
2. Related work using multiple projector-camera feedback sys-
tems. Murakami et al. [11] proposed another
2.1. Auto-stereoscopic display method for VDDAM based on reflectance meas-
urement among multiple projectors and cameras,
――――――――――――― which was equivalent to roughly sampled BRDF.
1
Corresponding author.
APMAR’22: The 14th Asia-PacificWorkshop on Mixed and Aug-
Amano and Yoshioka [12] combined reflectance
mented Reality, Dec. 02-03, 2022, Yokohama, Japan analysis with multiple projector-camera feedback
EMAIL: s226042@wakayama-u.ac.jp (Shogo Ohsumi); and expanded the applicable reflection property to
amano@wakayama-u.ac.jp (Toshiyuki Amano)
ORCID: 0000-0002-1393-8757 (Shogo Ohsumi); 0000-0003- retroreflection and improved robustness against
4146-1375 (Toshiyuki Amano)
© 2022 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ht
tp:
//
ceur
-ws
.or
g
Works
hop I
SSN1613-
0073
Pr
oceedi
ngs
environmental lighting changes. However, the pa-
rameters for anisotropy are unknown for Amano
et al. [10], whereas Murakami et al. [11] and
Amano and Yoshioka [12] must consider in ad-
vance what type of anisotropy was used to create
the target image.
In this paper, we propose a method to parame-
terize the VDDAM applying the Ashikhmin
BRDF model to the previously acquired reflection
characteristics. With this parameterization, we en-
hanced or reduced the anisotropy and then recre-
ated the reflectance. Finally, we calculated the
manipulation references for each viewing direc-
tion using the recreated reflectance. Figure 1: Experimental devices.
2.3. Reflectance matrix In this section, we introduce the Ashikhmin
BRDF model to fit the reflectance matrix. The
Murakami and Amano proposed a response Ashikhmin BRDF model is described by the sum
model for multiple projectors and cameras that of the specular and diffuse components, with the
considers color [11]. The RGB values at a point A specular component 𝜌= defined as follows:
in the captured and projected images are defined 𝜌! (𝐤" , 𝐤 # )
('!(𝐡𝐮)"+'#(𝐡𝐯)" )
as follows: =
((𝑛$ + 1)(𝑛% + 1) (𝐧𝐡) ("-(𝐡𝐧)" ) /
5𝑅 + (1 − 𝑅! ) 31 − (𝐤𝐡)4 8 , (7)
8𝜋 (𝐡𝐤) 𝑚𝑎𝑥3(𝐧𝐤" ), (𝐧𝐤 # )4 !
# % #
𝐂"! = $𝑐!" , 𝑐! , 𝑐!$ ' , 𝑐!" ≥ 0, 𝑐! ≥ 0, 𝑐!$ ≥ 0,
where 𝐤> and 𝐤 ? represent normalized vector to
𝑤ℎ𝑒𝑟𝑒 𝑖 = 1,2, … , 𝑢, (1)
the light and viewer. 𝐡 represent normalized half-
% vector between 𝐤> and 𝐤 ? . 𝐧 represent surface
7& = 8𝑝&" , 𝑝&# , 𝑝&$ :
𝐏
#
, 𝑝&" ≥ 0, 𝑝& ≥ 0, 𝑝&$ ≥ 0,
normal to macroscopic surface. 𝑛@ and 𝑛A repre-
𝑤ℎ𝑒𝑟𝑒 𝑗 = 1,2, … , 𝑣, (2) sent two phong-like exponents that control the
where 𝑢 denotes the number of cameras, and 𝑣 specular lobe shape. The larger the value of 𝑛@ ,
denotes the number of projectors. In this case, by the higher the directivity of reflection in the u di-
expressing the reflection at an object surface as a rection. Similarly, the larger the value of 𝑛A , the
7 ∈ ℛ'×' , it can be described as follows:
matrix 𝐾 higher the directivity of reflection in the v direc-
tion.
𝐂"! = 𝐾
7!& 𝑀!& 𝐏
7& , (3)
where the matrix 𝑀!& represents the color-mixing 3. Proposed method
matrix [13] that calibrates the color. Furthermore,
when multiple projectors or cameras are used, Our proposed method obtained a reflectance
they are represented as follows: matrix representing the optical response between
projectors and cameras corresponding to a
𝐂B = 𝐾
C𝑀C𝐏
C, (4)
roughly sampled BRDF on every single point on
where the object's surface with the experimental devices
" shown in Figure 1. Subsequently, we fitted the re-
𝐂" = $𝐂%!" , 𝐂%#" , … , 𝐂%$" ( , 𝐏
+ = $𝐏
,!" , 𝐏 ,$" (" ,
,#" , … , 𝐏 (5) flectance matrix with the Ashikhmin BRDF
𝑀!! 𝑀!" ⋯ 𝑀!# 6!!
𝐾 6!"
𝐾 ⋯ 6!#
𝐾 model and parameterized the reflectance relation-
* = ,𝑀"! 𝑀"" ⋯ 𝑀"# * ⎛ 6"!
𝐾 6""
𝐾 ⋯ 6"# ⎞
𝐾 ship. Then, the parameters were manipulated to
𝑀 0 ,𝐾 = . (6)
⋮ ⋮ ⋱ ⋮ ⋮ ⋮ ⋱ ⋮
𝑀$! 𝑀$" ⋯ 𝑀$# ⎝𝐾6$! 6$"
𝐾 ⋯ 6$# ⎠
𝐾
design a desired anisotropic reflection, yielding a
recreated reflection matrix. Finally, the VDDAM
Hereafter, we regard the color spaces as calibrated, based on the optical model achieved the desired
and we write 𝑀C𝐏C as 𝐏
C in the following sections. appearance that the reflectance matrix represented
by projecting images from multiple projectors.
2.4. Ashikhmin BRDF model[14]
3.1. Multiple projector-camera sys-
tems
In this paper, we employed 7 cameras (Ximea，
MQ013CGE2, resolution: 1280 × 1024 ) and 7
projectors (EPSON,EB-W05, resolution:1280 ×
800) in order to achieve high quality perceptual
BRDF manipulation with complex reflection
characteristics. The cameras and projectors were
placed in front of the target object and the other
projectors are placed radially at 15𝑑𝑒𝑔. intervals
around the projector 4 (Figure 1). The cameras are Figure 2: Operating object and an estimate of the 𝑛:
placed close to each projector. This arrangement and 𝑛< distributions.
takes into account the measurement and manipu-
lation of anisotropic reflections. where 𝑠 ∈ (𝑟, 𝑔, 𝑏), 𝑡 ∈ (𝑟, 𝑔, 𝑏), and 𝑘&'"# repre-
To obtain the reflectance matrix, first capture sents the reflectance of the 𝑠 color component of
the red projection from one projector with all cam- the 𝑡 color projection of projector 𝑗 captured by
eras. Similarly, the green and blue projection from camera 𝑖. 𝑛@=B , 𝑛A=B represent anisotropic scattering
one projector is captured by all cameras. This pro- for each direction, and u, v, 𝑅C=B , 𝑅==B represent the
cess is repeated with seven projectors. The reflec- intensities of the diffuse and specular components,
tance is obtained by dividing the image thus ac- respectively. Because the four parameters 𝑛@ , 𝑛A ,
quired by the RGB of the projection image. 𝑅= , and 𝑅C must be positive, we applied a non-
negative condition to the Levenberg-Marquardt
3.2. Manipulation Target Object method. This allows us to obtain the four parame-
ters of the Ashikhmin model for a single pixel
from the reflectance matrix at a single pixel.
We used a drawing foil of Nishijin silk textile,
Figure 2 shows the estimated 𝑛@ and 𝑛A . Be-
which contains patterns of birds, flowers, clouds,
cause there is no significant difference the color
and a mountain with rivers, as the manipulation
channel, this figure shows only the R channel.
target. Various threads, including gold thread and
Brightness expresses the value of each parameter,
dyed thread, are used in this Nishijin silk textile,
and the brighter area has a sharp specular reflec-
and differences in gloss can be seen, such as the
tion along each direction. A small difference be-
gold thread being more reflective than the dyed
tween the 𝑛@ and 𝑛A values indicates isotropy,
thread. The weaving method also causes differ-
whereas a large difference indicates anisotropy.
ences in reflectance characteristics. In the case of
Area (a) in the figure shows a twill weave using
twill weave, the ratio of warp to weft threads on
gold threads and has almost isotropic reflections.
the surface is close, resulting in isotropic reflec-
Area (b) is a satin weave that uses gold threads
tions. On the other hand, a satin weave has a
and exhibits strong anisotropy. Area (c) is a satin
higher ratio of warp threads than weft threads on
weave using dyed threads, and it has weak anisot-
the surface, resulting in anisotropic reflections. In
ropy. Area (d) is a twill weave using dyed threads
this study, we regard the target object as a plane.
and exhibits diffuse reflection.
3.3. Parameter estimation 3.4. Anisotropy manipulation
We employed the Levenberg-Marquardt
Our anisotropic manipulation aims to enhance
method, a nonlinear optimization scheme, and ob-
or reduce its reflection of the anisotropic reflec-
tained anisotropic reflection parameters by mini-
tion optically while maintaining the glossiness of
mizing the error function as follows:
the isotropic reflection. Based on this, we updated
𝐸(𝑛!"# , 𝑛$"# , 𝑅%"# , 𝑅""# ) the parameters as follows:
, *
"# 𝑛; = 𝑛: + 𝛼(𝑛: − 𝑛< ), 𝑛<; = 𝑛< (𝑛: ≥ 𝑛< )
= ) )(𝑘&' − 𝜌(𝐤(' , 𝐤 )& ; 𝑛!"# , 𝑛$"# , 𝑅%"# , 𝑅""# ))) , (8) ! : . (9)
;
&+( '+(
𝑛: = 𝑛: , 𝑛<; = 𝑛< + 𝛼(𝑛< − 𝑛: ) (𝑛: < 𝑛< )
(a) Anisotropic reduction (b) Anisotropic enhancement (a) Anisotropic reduction (b) Anisotropic enhancement
Figure 3:Target images. Figure 4: Projection images.
(a) Anisotropic reduction (b) Original appearance (c) Anisotropic enhancement
Figure 5: Projection results.
If 𝑛@ was greater than 𝑛A , the difference, which
is an anisotropy, was added to 𝑛@ with a multipli-
cation of the scaling factor α. Otherwise, the dif-
ference was added to 𝑛A as well. When 𝛼 > 1, the
difference was expanded, and anisotropy was en-
hanced. On the contrary, the anisotropy was com-
pletely removed when 𝛼 = −1.
We applied this manipulation to the estimated
parameters and obtained the desired target images
for the entire manipulation area.
3.5. Calculation of projection image (a) Satin weave (b) Twill weave
Figure 6: Variation of brightness for each viewpoint.
From the reflectance matrix and the target im-
ages 𝐂BB = (𝐂BB>
% B%
, 𝐂B? , … , 𝐂BB@
% %
) , we obtained the
projection images 𝐏 CB = (𝐏 CB>
% C% CB@
, 𝐏B? , … , 𝐏 % %
) for 4. Result
each projector. However, the projection images
CB must be positive. Therefore, in this paper, a
𝐏 The target images were created by updating 𝑛@
non-negative conditional optimization problem and 𝑛A , and by manipulating the value of 𝛼 in Eq.
min ∥ 𝐾C𝐏CB − 𝐂BB ∥? , (9), the projection images were obtained using Eq.
D (10), and then projected.
" # $ $
𝑤ℎ𝑒𝑟𝑒 𝑝B> ≥ 0, 𝑝B> ≥ 0, 𝑝B> ≥ 0, … , 𝑝BA ≥ 0. (10) Figures 3 and 4 shows examples of the target
and projection images, respectively. In Figure 5,
used the Lawson-Hanson algorithm [15] to obtain
the projection results are arranged to correspond
the projection value of each projector at a certain
to each camera position. All of these images are
point. This calculation was performed for all the
shown in identical aspects by geometrical trans-
points in the operating range to obtain the projec-
formation to the common coordinate (cam 4).
tion image for each projector.
4.1. Anisotropic enhancement even with anisotropic reduction in the region of
the gold thread twill weave.
Figure 3 shows the target images created using
𝛼 = 8. We solved the non-negative optimization 5. Conclusion
problem described in Section 3.3 and obtained the
projection images shown in Figure 4(b). The ma- In this paper, we propose a method for estimat-
nipulation results from the projection are shown ing the parameters of the Ashikhmin model from
in Figure 5(c). It should be noted that a significant the reflectance matrix. Furthermore, we propose a
difference in glossiness between viewing direc- parameter manipulation method that can enhance
tions along u and v was observed. Figure 6 shows and reduce anisotropy. Projection images were
the brightness change from cam1 to cam5 and calculated using non-negative conditional optimi-
cam3 to cam7 when the viewpoint is moved from zation. The projection results showed that anisot-
left to right. Figure 6(a) shows the average bright- ropy could be enhanced and reduced in areas with
ness of the 3 × 3 pixels in the area shown in Fig- anisotropic reflections. In areas with isotropic re-
ure 2(b). When the viewpoint is moved horizon- flections, although the brightness was slightly re-
tally in the range of cam1 to cam5 and cam3 to duced, the isotropic reflections were maintained.
cam7, the gloss change of the anisotropy en-
hanced image is sharper than that of the original References
appearance. These results confirmed the enhanced
anisotropy.
Figure 6(b) shows the average brightness val- [1] S. Shimazu et al., “3d high dynamic range
ues of the 3 × 3 pixels in the area shown in Figure display system”, ISMAR, pp.235–236,
2(a). When the viewpoint is moved horizontally (2011).
in the range of cam1 to cam5 and cam3 to cam7, [2] T. Amano, H. Kato, “Appearance control us-
it can be confirmed that the anisotropic enhance- ing projection with model predictive control”,
ment follows the brightness change of the original ICPR, pp.2832–2835, (2010).
appearance, although the brightness of the aniso- [3] T. Okazaki et al., “A projector-camera sys-
tropic enhancement is reduced compared to the tem for high-quality synthesis of virtual re-
original appearance. This confirms that the iso- flectance on real object surfaces”, IPSJ,
tropic reflection was maintained in the region of pp.71-83, (2010).
the gold thread twill weave. [4] A. J. Law et al., “Perceptually based appear-
ance modification for compliant appearance
editing”, CG Forum, 30(8), pp.2288-2300,
4.2. Anisotropic reduction (2011).
[5] W. Matusik, H. Pfister, “3D TV: a scalable
Figure 3(a) shows the target images created system for real-time acquisition, transmis-
with 𝛼 = −0.5. The projection images and ma- sion, and autostereoscopic display of dy-
nipulation results are shown in Figures 4(a) and namic scenes”, SIGGRAPH, 23(3), pp.814-
5(a), respectively. The overall images are dark- 824, (2004).
ened, however the central river area has lost its [6] R. Yang et al., “Toward the light field dis-
gloss. As shown in Figure 6(a), when the view- play: Autostereoscopic rendering via a clus-
point is moved horizontally in the range of cam1 ter of projectors”, IEEE TVCG, 14(1),
to cam5 and cam3 to cam7 in the region shown in pp.84–96, (2008).
Figure 2(b), the anisotropic reduction shows a [7] A. Jones et al., “An automultiscopic projec-
smaller brightness change than the original ap- tor array for interactive digital humans” SIG-
pearance. From these results, we can confirm the GRAPH, (2015).
reduction in anisotropy. [8] K. Nagano et al., “An autostereoscopic pro-
As shown in Figure 6(b), when the viewpoint jector array optimized for 3D facial display”,
is moved horizontally in the range of cam1 to SIGGRAPH, (2013).
cam5 and cam3 to cam7 in the region shown in [9] T. Amano, K. Minami, “Structural color dis-
Figure 2(a), the anisotropy reduction follows the play on retroreflective objects”, ICAT -
brightness change of the original appearance, alt- EGVE, pp. 37– 44, (2015).
hough the brightness of the anisotropy decreases
compared to the original appearance. This con-
firms that isotropic reflection was maintained
[10] T. Amano et al., “Viewpoint-Dependent Ap-
pearance-Manipulation with Multiple Pro-
jector-Camera Systems", ICAT-EGVE,
pp.101-107, (2017).
[11] K. Murakami, T. Amano, “Materiality Ma-
nipulation by Light-Field Projection from
Reflectance Analysis”, ICAT-EGVE, pp.99-
105, (2018).
[12] T. Amano, H. Yoshioka, “Viewing-Direction
Dependent Appearance Manipulation Based
on Light-Field Feedback”, EuroVR, pp.192-
205, (2020).
[13] S. K. Nayar et al. “A projection system with
radiometric compensation for screen imper-
fections”, PROCAMS, (2003).
[14] M. Ashikhmin, P. Shirley, “An anisotropic
phong BRDF model”, Journal of Graphics
Tools, (2000).
[15] C. L. Lawson, R. J. Hanson, “Solving least
squares problems”, (1974).