In this paper, we propose to extend the alignment correction method for moving projected objects to correction of 2-axis movement. We also propose a method to deal with afterimages that occur when a projection method using model predictive control is applied to a moving object. In this method, the projected image is generated considering the frame bufer of the camera and the alignment correction is applied. Experimental results show that the proposed method does not generate afterimages and that the intended projection is possible.
poses a method to correct alignment errors in projections from moving vehicles. However, the method was limited
Spatial Augmented Reality (SAR) has emerged as a to movement along a single axis, thus lacking support
technology that employs projectors to project virtual ob- for motion in other directions. In this study, we take a
jects into real world. This key technology, shader ramps, preliminary step towards supporting rotational motion
was proposed by Rasker et al[
In the realm of SAR research, Amano et al. introduced quently, compensating solely for the projection position
an innovative approach called as Appearance Control, was deemed insuficient to solve the problem.
Nonetheutilizing feedback processing through MPC(model predic- less, the Appearance Control with feedforward control
tive control) [
In a diferent context, Amano et al. proposed the ap- method [
The appearance control shown in Fig.1 is achieved by
the projector-camera feedback processing using MPC[
′ 2 ′Δ = {2 Δ − 2 } + 2
(1)
This section describes how to extend the direction of
movement of the projection target to two-axis. First, as
1. Capture image of the projection target using in Amano et al.’s method [
mum brightness projection). To extend this correspondence to 2-axis, we obtain 3. Estimate object appearance under white the pixel mapping by moving the projector-camera syslight projection using ˆ and the image ℎ tem along the X and Y axes of the Cartesian coordinate captured when projected white light. system, respectively. We denote the pixel mapping that 4. Apply arbitrary image processing on to gen- compensates for the positional deviation when moving erate target image . in the X-axis direction is 2Δ, and the pixel map5. Adjust with MPC from the diference between ping that compensates for the positional deviation in the and . Y-axis direction is 2Δ. 6. Convert the shape of to the projector coordi- The displacement of movement is calculated from nate system and projected it from the projector. these pixel mappings and extended to 2-axis by combining them. The pixel mapping 2 ′Δ which corrects the displacement when moving in 2-axis, can be obtained as follows. imFaigge.2anshdotwhse tphreojreecltaiotinonimshaipgebdeutwrienegnvtehheiccleapmtuorveed- 2 ′Δ = ′ {2Δ(, ) − (, ) } ment. In this figure, () and () represent the captured and projected images at time , respectively; + ′ {2Δ(, ) − (, ) } + (, ) and represent the captured and projected images by (2) the external projector-camera system used for calibra- where (, ) represent the coordinates of the captured tion, respectively. When the vehicle is moving at speed image before movement, , , ′ and ′ represent the , the annotation is projected to the forward position reference displacement of the X-axis and the Y-axis and with a displacement of ∆ for its processing latency of the arbitrary velocity in each axis direction, respectively. ∆ . Therefore, we can compensate for the misalignment 2(, ) and 2(, ) represent the coordidue to latency of the moving speed by finding the pixel nates of the corresponding captured image at the position mapping 2 Δ between () and ( + ∆ ). where the coordinates (, ) of the captured image be
When moving with a given velocity ′, we can com- fore the move have moved by the reference displacement pute the corresponding pixel mapping by linearly inter- ,. polating the pixel mapping 2 Δ at velocity as
When performing alignment correction, it is necessary to project to the position where the projection object has moved within the processing time of the system. In practice, however, there is not only the processing time, but also the delay due to the frame bufer that temporarily stores the images captured by the camera. Therefore, the images processed by the system are the images taken before the number of bufers, so when shifting the pro- Figure 3: Experimental setup jection position, the delay time due to the bufers also had to be considered.
However, this camera bufer is the cause and, with where (, ) are the coordinates of the generated (+
appearance control through MPC, simply shifting the 1). In addition, the apparent estimation under white
projected image cannot eliminate the resulting afterim- illumination, shown in processing step 2 of 2.1, uses the
age. We explain the causes in 3.2.1. following equation presented by Kono et al [
We denote time within the Appearance Control as . The projection taken in the captured image () is the projected image ( − ) before the number of bufers. This is the correct combination of images to use in the system’s calculations. However, since this system does not consider delays caused by bufers, the projection image ( − 1), which is one step ahead of the captured image (), is used in model prediction control. If the projection target is not moving, this combination can still produce the intended projection. However, even if the alignment correction method is applied to the projected image, if the projection target is in motion, the coordinates will no longer match due to the time lag. As a result, the diference image is not added at the intended position in process 5 described in section 2.1. Consequently, the diference image is projected as an afterimage.
To solve such problems, it is necessary to consider the delay caused by bufers. Therefore, we propose a method to realize Appearance Control using MPC without afterimages by considering bufers and applying alignment correction method.
The determination of the next projected image, which considers the bufer, is given by the following equation. ( + 1) ≈ ( − ) + ˆ− 1()(1 − )
{( + 1) − ()./( − 0)} where, is a tuning parameter and ./ represents elementwise division. By applying alignment correction method to the calculated projected image ( + 1), the projected image ′( + 1) onto the moving projection target is then determined as follows.
′( + 1) = 2 ′Δ(, ) ˆ() = [()./( − 0) ⊙ ( − ) + 0] (5) where ⊙ is element-wise multiplication.
Fig.3 shows our experimental setup. For the projectorcamera system, we used a BenQ MH550 with a resolution of 1600 × 900 as projector and a Ximea MQ013CG-E2 with a resolution of 1280 × 900 as camera. was mounted on a motorized linear stage to obtain pixel mapping compensated for displacement due to movement. The external projector for acquiring the pixel mapping was a Sony VPL-EW276 with a resolution of 1280 × 800 and the projector-camera system was positioned so that the range of the image captured by the projector fell within the projector’s projection range. The reference displacements for pixel mapping acquisition were set at 10 mm in the x-direction and 50 mm in the y-direction. Also, we determined the value of is 3, which minimizes misalignment, through empirical experience. Decimal values Obtained in calculations (2) were rounded to the nearest whole number.
To validate the efectiveness of the proposed method, we compared the appearance control with the proposed (3) method. Fig. 4 shows the projection results when the projection object is moved by a motor stage and at an arbitrary speed. In this evaluation we used color phase control for image processing. The processing time for this projection was approximately 42 fps. The results show that appearance control produces an afterimage. In contrast, the proposed method prevents afterimages from occurring while the projection target is moving and (4) shows overlapping projection results.
The results of 4.1 confirm the efectiveness of the proposed method. In this section, in a similar experimental environment, saturation enhancement was used for image processing, and saturation was compared in three states: with the projection target stationary, moved with the proposed method, and moved without the proposed method. As a projection target, we used a sheet of paper printed with a square of 60 mm in length and width. The projection target was moved at -30 mm/s in the Y-axis direction, and the image taken by the system at the position where the projection target was moved 100 mm was used for evaluation.
Fig.5 shows the experimental results. The results show that when the projection target is moved without using the proposed method, the saturation is reduced in the red square area compared to the state where the projection target is stationary. On the other hand, when the proposed method is used, it can be seen that the saturation has not changed from the stationary state. From this result, we can say that the proposed method is capable of the same projection even when the projection target is moving. However, as can be confirmed by carefully looking at the contour of the projected object in Fig.4, a misalignment in the projected position was observed.
We rounded of the small number of decimal places generated by the calculation in (2) to one decimal place as misalignment compensation. However, truncated and rounded values can accumulate as errors as the steps progress. As a solution to this problem, we are currently considering creating an error map of the decimal part of the result of Eq. (2). Using this map, the decimal portion is accumulated according to the processing steps, and the amount of displacement of pixels whose value exceeds 1 is adjusted. We expect that this method can eliminate the misalignment that occurs.
Also, the proposed method continues to generate afterimages when used in situations where the moving speed of the object is diferent from the speed given to the system. Therefore, it is necessary to consider methods that do not require providing advance information about the movement of the projection target to the system. We will address this problem in our future work.
In this study, a method is proposed to compensate for misalignment of the projection with respect to a moving projection object in any direction. The proposed method extends the compensation for misalignment in two axes to achieve accurate projection alignment. We have also proposed a method for intended projection on moving objects in Appearance Control with MPC. Experimental results showed that the proposed method did not produce afterimages when the object moved in any direction. However, it was also found that the projected positions did not perfectly match due to cumulative errors. In addition, when the proposed method was used in situations where the speed of the moving object was diferent from the speed given to the system, the proposed method continued to produce afterimages. These problems will be addressed in future studies.