Augmented reality as a tool for visualization of ultrasound propagation in heterogeneous media based on the k-space method

Abstract

For programming the AR tools, interactive objects and creating the markers, the method of fiber spaces (k-space) for modeling of ultrasonic wave propagation in an inhomogeneous medium using coarse grids, with maintaining the required accuracy was used. The algorithm and tools of augmented reality were introduced into the adaptive control system of the pulp gas phase in the iron ore flotation process using a control action on the basis of high-energy ultrasound dynamic effects generated by ultrasonic phased arrays. The tools of augmented reality based on k-space methods allow to facilitate wider adoption of ultrasound technology and visualize the ultra-sound propagation in heterogeneous media by providing a specific correspondence between the ultrasound data acquired in realtime and a sufficiently detailed augmented 3D scene. The tools of augmented reality allow seeing the field of ultrasound propagation, its characteristics, as well as the effect of the dynamic effects of ultrasound on the change in the gas phase during the flotation process.

Keywords

• Augmented reality
• ultrasound propagation
• k-space method

-

Nowadays, the growth of applications of augmented reality (AR) can be attributed to solutions, which allow to visualize products and their characteristics, add some interactive objects which allow looking inside the processes, etc.

Every year applications using augmented reality are gaining more and more popularity. It is used in various fields of activity: production, repair [20], training [21], sales [7], marketing, exhibitions, user guides [8], remote maintenance [17], and navigation [4]. To build a functioning system, a sufficiently powerful platform is needed, which can be a modern mobile device [9], due to their widespread and evergrowing capabilities. To draw virtual objects, markers are used that are located in the surrounding space. They are located and analyzed by special software [2; 6; 16].

For the control of the basic technological parameters and mineral beneficiation process control, an important task is to control the parameters of complex heterogeneous mediums, including solid, liquid and gas phases. ___________________ Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

In [10; 11; 12; 13; 14; 15] automatic control system of gas bubble size distribution based on the ultrasonic phased array technology, which allows to implement the efficient control of pulp gas phase composition, adjust the aeration degree, increase the flotation speed, increase the concentrate quality and energy efficiency of the entire mineral processing process is proposed. In this research, the ultrasound as the main tool was used. 2

Materials and methods To facilitate wider adoption of ultrasound technology we use the tools of augmentedreality to visualize the ultrasound propagation in heterogeneous media by providing a specific correspondence between the ultrasound data acquired in real-time and a sufficiently detailed augmented 3D scene. We have established a tablet-based system for visualizing the propagation of high-energy ultrasound in a heterogeneous medium in the process of froth flotation using augmented-reality techniques in conjunction with the streaming data about ultrasound characteristics provided by phased array based on k-space method. This system gives the operator visual feedback as to the location of the ultrasonic spot generated by the elements of the phased array, the characteristics of the ultrasound beam, and look inside the flotation tank.

For programming, the AR tools, interactive objects and creating the markers, the method of fiber spaces (k-space) for modeling of ultrasonic wave propagation in an inhomogeneous medium using coarse grids, with maintaining the required accuracy was used [1; 11; 19].

We describe the ultrasonic waves propagation depending on the mass conservation equations, momentum conservation law and the equation of state using the first order dual equations, which can be summarized as follows [1; 5] p  x, t  t    x  c2  x  v  x , t     x  p  x , t  ,   x  v  x , t  t

 p  x , t   0 , where p  x, t  – the time and space dependent ultrasound pressure perturbations (x – 3D Cartesian axis (x, y, z));   x  is the spatially dependent density; c  x  is the spatial dependent sound speed; v  x, t  is the velocity of the particle and   x  is the absorption coefficient which equivalent to the inverse of the relaxation time.

Let’s represent all absorption effects with one relaxation time. From ( 2 ), the simplified equation can be written as follows v  x , t   p  x , t    x  ( 1 ) ( 2 ) ( 3 ) We differentiate ( 1 ) with respect to time and variations in ( 2 ), and the final equation can be represented as follows 2 p  x , t  , Taking into account the permutations ( 4 ) By substituting this equation in ( 5 ), we obtain  t

   x  v  x , t        x  с2  x  p  x , t  t 

1   x  с2  x   2 p  x , t   t 2    x  p  x, t  2 p  x,t  с2  x  t  с2  x  t2  1   x 

p  x, t  p  x   2 p  x,t   0 , The simplification of the pressure deviation to the density gradient can be represented as follows

 p  x , t          x   2 p  x , t    x   p  x, t  p  x    x 2 Taking into account ( 7 ), eq. ( 8 ) can be represented as follows

 1      x 

 p  x, t    

1   x  с2  x  2 p  x, t  , ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) This is a linear wave equation of ultrasonic wave propagation in the heterogeneous medium with the absorption parameters.

Let’s simplify ( 9 ) by separating the parameters of the sound velocity c  x  and density   x  from the second derivatives of pressure taking into account the spatial and temporal variables to solve the problem of ultrasound propagation using the fiber space method.

The original equation can be written in the form  0 , The normalized pressure can be represented as follows By substituting this equation in ( 10 ) we obtain After simplifying

 1      x 

1  p 2  x , t   x , t    

 12  x    x  с2  x  2  x , t  2  x, t  Taking into account further simplifications the equation takes the form 2  x, t   с102 2t2x , t   с102 с02 12  x  2 12  x   x, t    с2с02x   2  x , t   1 t 2  Even more, simplification can be obtained by determining the functions q(r, t) and v(r, t) efficient sources, which can be summarized as follows By simplifying ( 11 ) we obtain q  x, t   с02 12  x   x , t  2  12  x  v  x, t    c2 сx02 , t  1  x, t 

 2  x, t   с102 2t2x ,t   с102  q  x,t   2v  x, t  

 , t 2  k 2 F  k,t   12 2 F  k,t   1  Q k,t   c0 t2 c02  2V  k,t  

 , t2  ( 11 ) ( 12 ) This equation can be easily transformed into the frequency domain by using the threedimensional spatial Fourier transform as follows where F(k, t), Q(k, t) and V(k, t) – three-dimensional spatial Fourier transformation of values   x , t  , q  x , t  and v  x, t  respectively. Equation ( 12 ) satisfies the total wavefield and is defined as the sum of the incident and scattered field   x, t   i  x, t   s  x,t  , and the scattered wave field. ( 13 ) For the case of an inhomogeneous medium, we introduce an additional source w x , t   s  x, t   v  x , t  and by substituting it into ( 13 ) we obtain the following expression 2W k, t  t 2

 k 2с02 W k, t  V k, t   Q k, t  ,  where V  k, t   F 1  с2  x    2   i  x , t   w  x , t  ; с0   Q k, t   с02 F    x 2 12  x    i  x , t   w  x , t   v  x , t   where F is a spatial Fourier transform.

Let’s use the substandard finite difference approach to solve this equation [10; 15]. Discretization of the time derivative gives

W  k, t  t   2W k, t   W k, t  t   4 sin2  с0kt     2    V k, t  W k, t   

Q k, t   с2k 2  0  , ( 14 ) Consider the wave equation on the grayscale for the fiber space method (k-space), which includes the non-linear characteristic of ultrasound, which can be represented as follows [10]: 2  x, t     x   x, t  2

1 1   x   с2  x  where  2  x, t  is the nonlinearity source,   x  is the nonlinearity coefficient. The harmonic oscillations equation can be represented as follows 2W 2  k , t   с2k 2  VNL2 k , t  W 2 k , t   Q k , t  , 0 ( 15 ) where w2  x, t   s  x, t   vNL2  x, t  – additional source; W 2  k , t  is a spatial Fourier transform. After the spatial Fourier transformation, the equation can be expressed as follows The introduction of the nonlinearity term in fiber space method makes it easier to calculate the actual relief temperature in heterogeneous large scale models.

Let’s simulate the ultrasonic pressure field propagation in a heterogeneous medium using k-Wave toolbox (Matlab) which is designed for time domain ultrasound simulations in complex media like heterogeneous pulp. The simulation functions of this software are based on the k-space method and are both fast and easy to use [3; 18].

The net pressure of all piezoelectric elements can be obtained by adding the effects of each source and written in the form

n Pnet  x, y, z    pi  x, y, z  .

i1 q  x, y, z    Pn2et  x, y, z   c

, I  x, y, z   p2  x, y, z  2 c , ( 16 ) ( 17 ) ( 18 ) where I(x, y, z) – intensity at the point (x, y, z), W/m2.

The results of the ultrasonic wave propagation through a heterogeneous medium with density ρ = 1500 g/m3, for source strength of 1 MPa and tone burst frequency of 1 MHz for 16-element, phased array with a focus distance of 20 mm are shown on Fig. 1. The central slice absorption distribution in grayscale as a background and the square of the pressure distribution on the surface of this background are shown. Due to attenuation, the useful power at the point (x, y, z) is given by [18] The total energy at a point (x, y, z) is given by The final pressure field (a), the maximum pressure (b) and standard pressure (c) of the beam are shown in Fig. 2. The transducer focus and sidelobes are visible. The linear cross-section of the focus in the x direction is shown in Fig. 4: 1) for the single source; 2) simulation by a k-space method in the water; 3) in a heterogeneous medium. Based on the obtained results, augmented reality tools (interactive objects, markers) (see Figure 5, 6) were developed in the AR Editor, which are implemented based on the following algorithm:  STEP 1. Image Capture o STEP 1.1 Connecting the camera o STEP 1.2 Capturing video from the camera o STEP 1.3 Reading video frame frames  STEP 2. Recognition of special points and descriptors o STEP 2.1 Finding special points on the image o STEP 2.2 Calculation ─ singular points descriptors  STEP 3. Comparison of descriptors calculated in STEP 2.2 with the database of marker descriptors. Getting id markers, point vector  STEP 4. Reproduction of relevant content o STEP 4.1. If the returned number of points in the vector is greater than the specified value, then go to STEP 4.2. Otherwise, in STEP 4.4. o STEP 4.2 Getting the content of the marker from the database o STEP 4.3 Display the contents of the marker on the image o STEP 4.4 Displaying the image on the screen The developed algorithm and tools of augmented reality were introduced into the adaptive control system of the pulp gas phase in the iron ore flotation process using a control action on the basis of high-energy ultrasound dynamic effects generated by ultrasonic phased arrays (see Figure 7). The adaptive control system based on the ultrasonic phased array is placed on the wall of the flotation machine. In accordance with the above method and the developed algorithm, the system generates a marker in which the current state of the system is embedded. The tools of augmented reality allow seeing the field of ultrasound propagation, its characteristics, as well as the effect of the dynamic effects of ultrasound on the change in the gas phase during the flotation process. Conclusions To build a model of the ultrasonic field in a randomly inhomogeneous medium, the fiber spaces method (k-space), which increased the accuracy of parameter estimation field is used. The tools of augmented reality based on k-space methods were developed, which allow to facilitate wider adoption of ultrasound technology and visualize the ultrasound propagation in heterogeneous media by providing a specific correspondence between the ultrasound data acquired in real-time and a sufficiently detailed augmented 3D scene.

References

1. Bergmann , L. : Der Ultraschall: Und Seine Anwendung in Wissenschaft Und Technik . S. Hirzel Verlag, Stuttgart ( 1954 )

2. Hruntova , T.V. , Yechkalo , Yu.V. , Striuk , A.M. , Pikilnyak , A.V. : Augmented Reality Tools in Physics Training at Higher Technical Educational Institutions . In: Kiv, A.E. , Soloviev , V.N. (eds.) Proceedings of the 1st International Workshop on Augmented Reality in Education (AREdu 2018 ), Kryvyi Rih, Ukraine, October 2, 2018 . CEUR Workshop Proceedings 2257 , 33 - 40 . http://ceur-ws. org/ Vol- 2257 /paper04.pdf ( 2018 ). Accessed 30 Nov 2018

3. Introduction to Phased Array Ultrasonic Technology Applications: Olympus Guideline. Olympus, Waltham ( 2017 )

4. Lvov , M.S. , Popova , H.V. : Simulation technologies of virtual reality usage in the training of future ship navigators . In: Kiv, A.E. , Shyshkina , M.P . (eds.) Proceedings of the 2nd International Workshop on Augmented Reality in Education (AREdu 2019 ), Kryvyi Rih, Ukraine, March 22 , 2019 , CEUR-WS.org, online ( 2020 , in press)

5. Mast , T.D., Souriau , L.P. , Liu , D.-L.D. , Tabei , M. , Nachman , A.I. , Waag , R.C. : A k-Space Method for Large-Scale Models of Wave Propagation in Tissue . IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 48 ( 2 ), 341 - 354 ( 2001 ). doi: 10 .1109/58.911717

6. Meisner , J. , Donnelly , W.P. , Roosen , R.: Augmented reality technology . US Patent 6 , 625 , 299 B1 , 23 Sept 2003

7. Mintii , I.S. , Soloviev , V.N. : Augmented Reality: Ukrainian Present Business and Future Education . In: Kiv, A.E. , Soloviev , V.N. (eds.) Proceedings of the 1st International Workshop on Augmented Reality in Education (AREdu 2018 ), Kryvyi Rih, Ukraine, October 2, 2018 . CEUR Workshop Proceedings 2257 , 227 - 231 . http://ceur-ws.org/Vol2257/paper22.pdf ( 2018 ). Accessed 30 Nov 2018

8. Modlo , Ye .O., Semerikov , S.O. , Bondarevskyi , S.L. , Tolmachev , S.T. , Markova , O.M. , Nechypurenko , P.P. : Methods of using mobile Internet devices in the formation of the general scientific component of bachelor in electromechanics competency in modeling of technical objects . In: Kiv, A.E. , Shyshkina , M.P . (eds.) Proceedings of the 2nd International Workshop on Augmented Reality in Education (AREdu 2019 ), Kryvyi Rih, Ukraine, March 22 , 2019 , CEUR-WS.org, online ( 2020 , in press)

9. Modlo , Ye .O., Semerikov , S.O. , Nechypurenko , P.P. , Bondarevskyi , S.L. , Bondarevska , O.M. , Tolmachev , S.T.: The use of mobile Internet devices in the formation of ICT component of bachelors in electromechanics competency in modeling of technical objects . In: Kiv, A.E. , Soloviev , V.N. (eds.) Proceedings of the 6th Workshop on Cloud Technologies in Education (CTE 2018 ), Kryvyi Rih, Ukraine, December 21 , 2018 . CEUR Workshop Proceedings 2433 , 413 - 428 . http://ceur-ws. org/ Vol- 2433 /paper28.pdf ( 2019 ). Accessed 10 Sep 2019

10. Morkun , V. , Morkun , N. , Pikilnyak , A. : Modeling of ultrasonic waves propagation in inhomogeneous medium using fibered spaces method (k-space) . Metallurgical and Mining Industry 2 , 43 - 48 ( 2014 )

11. Morkun , V. , Morkun , N. , Pikilnyak , A. : The adaptive control for intensity of ultrasonic influence on iron ore pulp . Metallurgical and Mining Industry 6 , 8 - 11 ( 2014 )

12. Morkun , V. , Morkun , N. , Pikilnyak , A. : Ultrasonic phased array parameters determination for the gas bubble size distribution control formation in the iron ore flotation . Metallurgical and Mining Industry 3 , 28 - 31 ( 2014 )

13. Morkun , V. , Morkun , N. , Pikilnyak , A. : Ultrasonic testing of pulp solid phase concentration and particle size distribution considering dispersion and dissipation influence . Metallurgical and Mining Industry 1 , 9 - 13 ( 2015 )

14. Pikilnyak , A. : Adaptive control system of the iron ore flotation using a control action based on high-energy ultrasound . Metallurgical and Mining Industry 2 , 27 - 30 ( 2015 )

15. Pikilnyak , A. : The gas bubble size parameters monitoring and control method . Metallurgical and Mining Industry 7 , 19 - 21 ( 2015 )

16. Striuk , M.I. , Semerikov , S.O. , Striuk , A.M.: Mobility: a systems approach . Information Technologies and Learning Tools 49 ( 5 ), 37 - 70 ( 2015 ). doi: 10 .33407/itlt.v49i5. 1263

17. Syrovatskyi , O.V. , Semerikov , S.O. , Modlo , Ye.O. , Yechkalo , Yu.V. , Zelinska , S.O. : Augmented reality software design for educational purposes . In: Kiv, A.E. , Semerikov , S.O. , Soloviev , V.N. , Striuk , A.M. (eds.) Proceedings of the 1st Student Workshop on Computer Science & Software Engineering (CS&SE@SW 2018 ), Kryvyi Rih, Ukraine, November 30 , 2018 . CEUR Workshop Proceedings 2292 , 193 - 225 . http://ceur-ws.org/Vol2292/paper20.pdf ( 2018 ). Accessed 21 Mar 2019

18. Tabei , M. , Mast , T.D., Waag , R.C. : A k-space method for coupled first-order acoustic propagation equations . Journal of the Acoustical Society of America 111 ( 1 ), 53 - 63 ( 2002 ). doi:10.1121/1 .1421344

19. Ul 'trazvuk: Malaja jenciklopedija (Ultrasound: Small Encyclopedia) . Soviet Encyclopedia , Moscow ( 1979 )

20. Zelinska , S.O. , Azaryan , A.A. , Azaryan , V.A. : Investigation of Opportunities of the Practical Application of the Augmented Reality Technologies in the Information and Educative Environment for Mining Engineers Training in the Higher Education Establishment . In: Kiv, A.E. , Soloviev , V.N. (eds.) Proceedings of the 1st International Workshop on Augmented Reality in Education (AREdu 2018 ), Kryvyi Rih, Ukraine, October 2, 2018 . CEUR Workshop Proceedings 2257 , 204 - 214 . http://ceur-ws.org/Vol2257/paper20.pdf ( 2018 ). Accessed 30 Nov 2018

21. Zinonos , N.O. , Vihrova , E.V. , Pikilnyak , A.V. : Prospects of Using the Augmented Reality for Training Foreign Students at the Preparatory Departments of Universities in Ukraine . In: Kiv, A.E. , Soloviev , V.N. (eds.) Proceedings of the 1st International Workshop on Augmented Reality in Education (AREdu 2018 ), Kryvyi Rih, Ukraine, October 2, 2018 . CEUR Workshop Proceedings 2257 , 87 - 92 . http://ceur-ws. org/ Vol- 2257 /paper10.pdf ( 2018 ). Accessed 30 Nov 2018