Methods for the synthesis of realistic images formed by optical devices containing hologram optical elements

Abstract

Nowadays, virtual tests of optical systems with holographic elements are being used more and more. Despite the fact that holographic optical elements have existed for decades and most programs for designing optical systems include ray tracing modules in optical systems containing holographic elements, the synthesis of realistic images formed by these systems is absent in most of these programs. The paper presents the results of a study of the possibility of implementing an effective and physically accurate stochastic ray tracing through hologram optical elements. The theoretical foundations of the light propagation through hologram optical elements are considered and a detailed ray tracing algorithm for its implementation in forward, backward and bidirectional stochastic ray tracing methods are presented. The results of modeling the propagation of rays and the synthesis of a realistic image formed by a two-hologram augmented reality system are presented. Also, we conducted a study of the influence of the observer's eye pupil position on the quality of the formed image.

1. Introduction

Even though hologram optical elements have existed for decades and most optical system design programs [1], for example, CODE-V [2] or Zemax [3] incorporate ray tracing modules in optical systems containing hologram optical elements, the synthesis of realistic images generated by these optical systems is absent in most of these programs. Of particular difficulty is the task of designing optical systems with non-deterministic beam propagation. For example, the problem of synthesizing images formed by light guide systems containing holographic optical elements.

Historically, hologram optical elements were used not only as separate optical devices but also as elements of complex devices. The first experiments with hologram elements (zone plates) [4] were carried out more than 100 years ago, and even then it was noted that the zone plate can form a high-resolution image corresponding to a lens with a similar aperture, and it has strong chromatism, which can be used for the correction of chromatic aberrations of optical systems [5]. Accordingly, to solve the problems of designing optical systems with hologram elements, a methodology was developed for calculating the aberrations of the third [6, 7] and higher orders [8]. With the advent of high-performance computers, methods of ray tracing through hologram optical elements were developed and introduced into optical design systems [912]. However, all the proposed solutions concerned mainly the methods of deterministic ray tracing, calculation of aberrations and image quality functions (point spread function and optical transfer function). All the effects associated with diffraction scattering were solved separately, and this did not allow us to evaluate how diffraction scattering affects the quality of the formed image. Moreover, software solutions did not allow for “full-scale” virtual tests of optical systems with hologram optical elements, when the object is not a point or the resolution test target, but a real three-dimensional scene.

At present, virtual tests of optical systems with hologram optical elements are increasingly used. This is due to increased requirements for the quality of the image formed by optical systems containing elements of this type. This is also due to the emergence of a new type of device in which hologram optical elements are used in an “unconventional” manner. Such devices include augmented and mixed reality systems in which an image formed on a liquid crystal screen is transmitted to the observer through light-conducting devices containing phase diffraction structures. A good example of such a device is the three-hologram system of a HoloLens mixedreality device [13]. Another example of the use of hologram optical elements can be dynamic phase modulators used in virtual reality systems to eliminate the discomfort of visual perception caused by the mismatch of vergence and accommodation of human vision that occurs when observing a flat image on a CCD matrix [14-16].

Along with all the advantages that hologram optical elements provide for creating high-resolution images, they have several significant drawbacks. This is mainly diffraction scattering, caused by technological problems in the manufacture of these elements and the use of hologram elements in wide spectral regions, leading to a shift in the maximum diffraction efficiency to neighboring diffraction orders. Low diffraction efficiency and diffraction scattering can reduce the overall image quality and require special methods for calculating and synthesizing images that take into account all the effects associated with diffraction scattering and image formation by the optical system.

In this paper, we consider stochastic ray tracing methods in optical systems containing hologram optical elements, as well as methods for synthesizing images formed by these systems with allowance for diffraction scattering effects on hologram optical elements. 2.

Materials and method

The hologram optical element is an interference pattern structure recorded on the recording material in the form of modulation of the refractive index, transmission or surface relief. Since the conditions of applicability of geometric optics are violated when an electromagnetic wave passes through a holographic element, an approach based on recording the interference pattern on its “sensitive” surface, specified as a three-dimensional function ( ⃗ ), is used. The interference structure pattern formation is shown in Fig. 1, where two sources (the object source O and reference source R) of coherent radiation with the corresponding wavefronts and form the interference pattern on the surface ( ⃗ ). It should be noted that the recording scheme is shown in Fig. 1, can exist only in the form of a mathematical model, and the corresponding holographic structure can be manufactured according to the calculation results for this model. However, this figure shows the principle of recording and reconstruction of the holographic element. If the surface ( ⃗ ) is located outside the caustic region of the reference and reference light sources, then the optical path from these sources is described by the corresponding eikonal equations: ( ⃗ , ⃗ ) and ( ⃗ , ⃗ ) or ( ⃗ , ⃗ ) and ( ⃗ , ⃗ ). The last two eikonals refer to the case of infinitely distant light sources, where ⃗ and ⃗ are the directions of propagation of the corresponding plane waves. Let us consider how the light wave is converted by thin holograms (both recorded and synthesized). As a result of recording or synthesizing a hologram, a change in the transmittance, refraction, or its relief occurs. For this, the concept of the characteristic function of a hologram is introduced: ( 1 ) = − According to the eikonal equation:

∇ = ⃗ − ⃗ ( 2 ) where and are the refractive indices of the media of the object and reference light sources.

Then, the so-called diffraction grating vector can be written as:

⃗ = 10 ∇ ( 3 ) where 0 is the wavelength of the hologram recording.

The neighborhood of each point of the hologram recorded by unfocused beams can be represented as a diffraction grating, the frequency, and orientation of which is a function of the position of the point on the surface of the hologram. The given frequency and orientation of the local “diffraction grating” determines the law of transformation of rays on the surface of the hologram.

If ⃗ is designated as a vector of local spatial frequencies on the surface of a hologram, i.e. the projection of the vector ⃗ onto the plane tangent to the surface of the hologram at the point of incidence of the beam, then the rays transformation law can be written as: ( 4 ) ⃗′ = ⃗ + ⃗ + ⃗ where ⃗ is the direction of the ray incident on the hologram surface, ⃗′ is diffracted ray direction, is diffraction order, is the incident ray wavelength, ⃗ is the direction of the local normal to the hologram surface at the point of the ray incidence, is the coefficient determined from the condition ⃗′ = 1.

Taking into account the fact that the rays ⃗′ and ⃗ can be in different media, it is necessary to take into account the law of refraction, according to which expression 4 can be written in the form:

′ ⃗′ = ⃗ + ⃗ + ⃗ ( 5 )

This expression determines the law of the ray transformation on the surface of the hologram in each diffraction order. However, this expression does not allow us to understand what fraction of the light energy will go in each diffraction order.

A general solution for calculating the diffraction efficiency of a hologram element can be obtained numerically from the Maxwell equations. However, this solution allows us to calculate the state of diffracted electromagnetic fields at given points in space, which is not an analog of rays in the geometric approximation. Therefore, for each of the types of hologram optical elements (high-frequency diffraction structures like diffraction gratings, volumetric multilayer holograms, thin phase holograms (kinoforms), the most suitable solutions are sought that allow one to calculate diffraction efficiency. In optical devices, as a rule, thin phase structures such as diffraction gratings and kinoforms are used. Special solutions [17] are used to calculate the efficiency of diffraction gratings. For kinoform elements, the complex function of diffraction efficiency is the amplitude factor of the eikonal of the light radiation passing through the holographic structure:

( , ) = ∑ , ( , ) , ( , ) ( 6 ) where ( , ) is the pupil function of kinoform (or optical system with kinoform), , ( , ) is the diffraction efficiency of kinoform, , ( , ) is the eikonal, is the wavenumber.

For hologram structures made in the form of a linearly varying phase relief on the surface of a material with a refractive index n, the distribution of diffraction efficiency has the following form: , ( , ) = (−1) ( − ) ( 7 ) where = −1 , is the index of refraction of the −1 medium of hologram relief for the working wavelength of the hologram, is the index of refraction of the medium of hologram relief for the recording wavelength of the hologram.

Schematically, this relief (kinoform) is presented on the left side of Fig. 2. Based on technological considerations, kinoforms are usually approximated by a set of plane-parallel steps. The right part of fig. 2 illustrates this relief. The vertical axis of the graphs corresponds to the phase incursion, the horizontal axis to the period of the kinoform relief reduced to 2 . In this case, the diffraction efficiency has the following form: , ( , ) = (−1) ∙ ( − ) − (8) where N is the number of quantization levels of the kinoform relief phase. scattering function and the transfer function by the following relationships: (9) ℱ{ } = 1 ⋅ where ℱ is two-dimensional Fourier transform operator, distribution in the image plane, is the point spread function, is the amplitude – optical transfer function, 1

and 2 Obviously,

the distribution of the are normalization factors.

diffraction intensity efficiency

for of light radiation the is proportional to the square of the modulus of the complex function of diffraction efficiency ( 7 ) and (8).

As a rule, kinoform elements in optical devices are designed for the same diffraction order and in this order, their diffraction efficiency should be maximum. However, expression ( 7 ) demonstrates a decrease in diffraction efficiency when the wavelength deviates from the wavelength for which the kinoform was calculated.

Therefore, its maximum efficiency can be ensured only at the design wavelength. Besides, the diffraction efficiency is highly dependent on the quality of the approximation of the kinoform relief. With a decrease in the number of quantization levels of the relief phase (8), its diffraction efficiency drops sharply.

Ray tracing algorithms and luminance calculation

The above expressions allow ray tracing to be implemented in optical systems containing hologram optical elements. For a given diffraction order, the deterministic and stochastic ray tracing has a common ray transformation method on a hologram optical element ( 5 ). The difference lies in the fact that in the first case the diffraction order is set forcibly, and in the second case, it is chosen randomly, for example, with a probability density equal to the diffraction efficiency of the hologram optical element.

For ray tracing, two main methods for specifying a hologram element are used. Firstly, this is the explicit specification of its characteristic equation ( 2 ) and, secondly, this is the specification of the characteristic equation in the form of a recording scheme shown, for example, in Fig. 1. The recording scheme can exist in the form of two point light sources forming an interference pattern at the point of incidence of the ray on the surface of the hologram

optical element, or optical systems projecting point light sources on this surface. In the latter case, it is necessary to find the optical path, for example, by the method of fitting the rays, from the light source to the point of incidence of the ray on the surface of the hologram optical elements. All methods for specifying a hologram optical element and tracing deterministic rays through these elements are known and described in the literature [9 -12].

Therefore, special attention should be paid to stochastic ray tracing methods and methods of ray tracing for holograms recorded from an extended reference light source that simulates the scattering of light by scene objects. To simulate the process of stochastic scattering, it is supposed to use the method of sampling the diffraction order by its significance. This means that the probability density of diffraction scattering for different diffraction orders will be proportional to the square of the diffraction efficiency modulus of the hologram optical element. It should be noted that diffraction scattering on a hologram element is fundamentally different from light scattering on a simple diffuse object. If for a diffuse surface the bidirectional scattering function is continuous and is defined as the brightness coefficient for given directions of observation and illumination, then for diffraction scattering this function is discontinuous and is defined as the transmittance of the power of light radiation for a given diffraction order. Therefore, from the computer simulation process, the diffraction scattering function is “specular”, i.e. redistributes light radiation in directions corresponding to given diffraction orders ( 5 ). Also, even for a single diffraction order, a light beam that transfers energy in a wide region of the spectrum that after scattering is converted into a beam of monochromatic rays propagating in the directions defined by formula ( 5 ). The energy of scattered rays is determined by diffraction efficiency. To calculate the diffraction efficiency of thin phase plates, one can use formulas ( 7 ) and (8). In general, the diffraction efficiency is a function of wavelength, diffraction order, polarization state, and direction of incidence of light radiation. Also, diffraction efficiency depends on the manufacturing technology of diffraction elements and may vary along its surface. Therefore, the calculation of diffraction efficiency is a separate task, which, in most cases, can be solved using specialized software [18, 19]. After determining the distribution of diffraction efficiency over the surface of the hologram, the corresponding multidimensional integral of the probability of diffraction scattering is calculated. In the simplest case, this is a fourdimensional integral of the diffraction order, wavelength, and two angles of the direction of the light ray incidence. This integral is a multidimensional tabular function, the nodes of which are fixed beam parameters, for example, a set of beam incidence angles for which diffraction efficiency was calculated.

The stochastic ray tracing algorithm is the following: ( 1 ) Search for the point of intersection of the ray with the surface of the hologram optical element. ( 2 ) Calculation of the eikonal of the hologram at the point of incidence of the beam. For a hologram specified in the form of a characteristic equation or simple point (object and reference) light sources, the calculation is performed by the shadow ray tracing method. If point sources are used in conjunction with optical systems, the ray fitting method is used, which finds the optical path from the source to the point of intersection. If the light source is extended - there is equal-intensity random sampling from the point of intersection of the beam to the surface of the light source and for the found pair of points, the eikonal of the hologram is calculated. ( 3 ) Transition to monochromatic representation. If the incident beam contains radiation in a wide region of the spectrum, then the wavelength for which diffraction scattering will be performed is selected. The choice is made randomly by the significance of the density of the spectral radiation of the incident ray. After the ray has been converted into a monochromatic representation, the order of diffraction scattering is chosen. It should be noted that the choice of the wavelength of monochromatic radiation is carried out from the list of wavelengths for which the diffraction efficiency of the hologram element was calculated. And if the optical system contains more than one hologram element, then their diffraction efficiency should be determined for the common list of wavelengths. ( 4 ) The choice of the direction of the ray scattering. For a given wavelength, the probability integral of diffraction scattering is singled out. Further, by the method of bilinear interpolation weights are selected that determines the proximity of the ray parameters at the point of incidence to the nodes of the probability function of diffraction scattering. Then, using the method of choosing the significance between the obtained weights, a local function of diffraction efficiency, which depends only on the diffraction order, is randomly distinguished. Using the selection method by significance, the diffraction order is randomly selected, which is further used in formula ( 5 ) to calculate the direction of the diffracted beam. If the probability of absorption of a light ray was included in the probability function of diffraction scattering, then the absorption event leading to a halt in its propagation is automatically realized when the diffraction order is chosen. Since the probability density of all events coincided with the diffraction efficiency of the hologram optical element, the energy of the light ray will not change as a result of its transformation. ( 5 ) If an absorption event has not occurred, then the ray continues to propagate in the system with the same energy.

This algorithm is consistent with ray tracing software interfaces that exist in most optical modeling and physically correct rendering systems. In the above algorithm, the hologram optical element is a “specular” energy converter and does not form its luminance. When using hologram elements in the rendering system, it is necessary to understand that this element does not form its secondary luminance, but allows you to transmit luminance when observing or lighting through it, which allows you to use it in all photorealistic image synthesis programs based on forward, backward and bidirectional tracing methods rays. 4.

Results

The ray tracing method for scenes with hologram optical elements was implemented as part of the Lumicept computer-based system for the synthesis of photorealistic images [20]. As a hologram optical element, a thin phase plate with relief variation (kinoform) was supported. The diffraction efficiency was calculated using formulas ( 7 ) and (8) and was a four-dimensional function of the wavelength, diffraction order, and direction of the light radiation incidence (the direction of incidence was modeled as a change in the depth of the phase relief).

As an example, a scene was constructed that included a two-holographic augmented reality optical system projecting an image from an LCD matrix through a light guide system into the eye of an observer. The characteristic equations of the holograms were set analytically, based on the conditions for the input of light from the projection system of the LCD matrix into the light guide plate at an angle of total internal reflection, and then the radiation was output by the second hologram to the eye pupil of the observer. The diffraction efficiency of these holograms was calculated outside the Lumicept software package and set in tabular form. The schematic diagram of this device is shown in Fig. 3. The field of view of this system is 19°. For the synthesis of images, the forward stochastic ray tracing method was used.

The simulation result is shown in Fig. 4. Modeling was performed for nine regions of the position of the pupil of the observer's eye relative to the light guide plate. It is seen that in the lower and far zones of the light guide plate, a decrease in image brightness is observed, up to the complete disappearance of the image. This suggests the necessity to optimize the distribution of the diffraction efficiency of holograms over the surface of the light guide plate.

5. Conclusion

In the framework of this study, an effective and physically correct method of ray tracing through hologram optical elements was proposed. To implement photorealistic rendering in a system, it is enough to simply support this algorithm in the existing software interface of the optical properties of the surface. The proposed algorithm is applicable for forward, backward and bidirectional stochastic ray tracing methods.

The main tasks that are planned to be performed as part of the expansion of the proposed approach are the development and implementation of methods for calculating diffraction efficiency for a wider range of hologram optical elements, mainly for high-frequency diffraction gratings and phase holograms with a nonstandard shape of the phase relief of the hologram substrate, given, for example, ambiguous function.

Asknowledges

The work was supported by RFBR, Grants № 18-0801484 and 19-01-00435. [8] P.C. Mehta (1974) Fifth-order Aberrations in in-line Holograms, Optica Acta: International Journal of Optics, 21:12, 1005-1008, DOI: 10.1080/713818870 [9] Mikhail A. Gan, Dmitriy D. Zhdanov, Vadim V.

Novoselskiy, Sergey I. Ustinov, Alexander O. Fedorov, Igor S. Potyemin, and Sergey N. Bezdidko "Design of optical systems with HOE by DEMOS program", Proc. SPIE 1574, Intl Colloquium on Diffractive Optical Elements, (1 October 1991); https://doi.org/10.1117/12.50131 [10] Mikhail A. Gan, Dmitriy D. Zhdanov, Vadim V.

Novoselskiy, Sergey I. Ustinov, Alexander O. Fedorov, and Igor S. Potyemin "DEMOS: state-ofthe-art application software for design, evaluation, and modeling of optical systems," Optical Engineering 31( 4 ), (1 April 1992). https://doi.org/10.1117/12.56130 [11] Mikhail A. Gan, Dmitriy D. Zhdanov, Vadim V.

Novoselskiy, Sergey I. Ustinov, Alexander O. Fedorov, and Igor S. Potyemin "DEMOS: new possibility for design and modeling of complex optical systems", Proc. SPIE 1780, Lens and Optical Systems Design, 17802F (15 April 1993); https://doi.org/10.1117/12.142804 [12] Mikhail A. Gan, Dmitriy D. Zhdanov, Vadim V.

Novoselskiy, Sergey I. Ustinov, Alexander O. Fedorov, and Igor S. Potyemin "DEMOS: tool for design and modeling of IR optical systems", Proc. SPIE 2106, CIS Selected Papers: Iconics and Thermovision Systems, (11 November 1993); https://doi.org/10.1117/12.163677 [13] US Patent: US 9,372,347 B1 [14] Matsuda, N., Fix, A. and Lanman, D., “Focal Surface Displays,” ACM Trans. Graph. 36, 4, Article 86, July, 14 pages (2017) [15] Potemin, I.S., Livshits, I., Zhdanov, D., Zhdanov, A. and Bogdanov N., "An application of the virtual prototyping approach to design of VR, AR, and MR devices free from the vergence-accommodation conflict," Proc. SPIE 10694, 28 May 2018, Computational Optics II, 1069404 (2018). [16] Igor S. Potemin, Andrey Zhdanov, Nikolay Bogdanov, Dmitry Zhdanov, Irina Livshits, Yan Wang, "Analysis of the visual perception conflicts in designing mixed reality systems," Proc. SPIE 10815, Optical Design and Testing VIII, 108150U (5 November 2018); doi: 10.1117/12.2503397 [17] Frank Wyrowski, Efficiency of quantized diffractive phase elements, Optics Communications 92 (1992), p,119-126 [18] F. Wyrowski and C. Hellmann, ’Geometrical Optics Reloaded – Physical optics modelling with smart rays’, Optik&Photonik, 43-47, 5/2015 [19] Wyrowski VirtualLab Fusion, unified optical design software developed by Wyrowski Photonics UG, distributed and supported by LightTrans GmbH, Jena, Germany, URL: http://www.wyrowskiphotonics.com/virtuallab/news.html (date of call 12.04.2020) [20] Hybrid Light Simulation Software Lumicept // Integra.jp. URL:

References

[1] Bernard C. Kress , Patrick Meyrueis, Applied digital optics : from micro-optics to nanophotonics , John Wiley & Sons, Ltd, 2009

[2] CODE V Optical Design Software // Synopsys, Inc., URL: https://www.synopsys.com/opticalsolutions/codev. html (date of call 06.04 . 2020 )

[3] OpticStudio // ZEMAX LLC, URL: https://www.zemax.com/products/opticstudio (date of call 06.04 . 2020 )

[4] Wood R.W. Phase-Reversal Zone Plates , and Diffraction-Telescopes , Phil. Magazine, 1898 , Vol. 45 , Ser . 5 , p. 511 - 522

[5] Miyamoto K. The Phase Fresnel Lens . - JOSA , 1961 , vol. 51 , No. 1 , p. 21 - 24

[6] Meier R. Magnification and third-order aberration in holography . - JOSA , 1965 . Vol. 55 , No 8, p987 - 992

[7] Champagne E.B. Nonparaxial Imaging, magnification and aberration properties in holography . -JOSA , 1967 , vol. 57 , No 1, p. 51 - 55 https://integra.jp/en/products/lumicept (date of call 10.03 . 2020 )