=Paper=
{{Paper
|id=Vol-2547/paper06
|storemode=property
|title=Augmented reality as a tool for visualization of ultrasound propagation in heterogeneous media based on the k-space method
|pdfUrl=https://ceur-ws.org/Vol-2547/paper06.pdf
|volume=Vol-2547
|authors=Vladimir S. Morkun,Natalia V. Morkun,Andrey V. Pikilnyak
|dblpUrl=https://dblp.org/rec/conf/aredu/MorkunMP19
}}
==Augmented reality as a tool for visualization of ultrasound propagation in heterogeneous media based on the k-space method==
https://ceur-ws.org/Vol-2547/paper06.pdf
81
Augmented reality as a tool for visualization of
ultrasound propagation in heterogeneous media based on
the k-space method
Vladimir S. Morkun[0000-0003-1506-9759], Natalia V. Morkun[0000-0002-1261-1170]
and Andrey V. Pikilnyak1[0000-0003-0898-4756]
Kryvyi Rih National University, 11, Vitaliy Matusevych Str., Kryvyi Rih, 50027, Ukraine
pikilnyak@gmail.com
Abstract. For programming the AR tools, interactive objects and creating the
markers, the method of fiber spaces (k-space) for modeling of ultrasonic wave
propagation in an inhomogeneous medium using coarse grids, with maintaining
the required accuracy was used. The algorithm and tools of augmented reality
were introduced into the adaptive control system of the pulp gas phase in the iron
ore flotation process using a control action on the basis of high-energy ultrasound
dynamic effects generated by ultrasonic phased arrays. The tools of augmented
reality based on k-space methods allow to facilitate wider adoption of ultrasound
technology and visualize the ultra-sound propagation in heterogeneous media by
providing a specific correspondence between the ultrasound data acquired in real-
time and a sufficiently detailed augmented 3D scene. The tools of augmented
reality allow seeing the field of ultrasound propagation, its characteristics, as well
as the effect of the dynamic effects of ultrasound on the change in the gas phase
during the flotation process.
Keywords: Augmented reality, ultrasound propagation, k-space method.
1 Introduction
Nowadays, the growth of applications of augmented reality (AR) can be attributed to
solutions, which allow to visualize products and their characteristics, add some
interactive objects which allow looking inside the processes, etc.
Every year applications using augmented reality are gaining more and more
popularity. It is used in various fields of activity: production, repair [20], training [21],
sales [7], marketing, exhibitions, user guides [8], remote maintenance [17], and
navigation [4]. To build a functioning system, a sufficiently powerful platform is
needed, which can be a modern mobile device [9], due to their widespread and ever-
growing capabilities. To draw virtual objects, markers are used that are located in the
surrounding space. They are located and analyzed by special software [2; 6; 16].
For the control of the basic technological parameters and mineral beneficiation
process control, an important task is to control the parameters of complex
heterogeneous mediums, including solid, liquid and gas phases.
___________________
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
82
In [10; 11; 12; 13; 14; 15] automatic control system of gas bubble size distribution
based on the ultrasonic phased array technology, which allows to implement the
efficient control of pulp gas phase composition, adjust the aeration degree, increase the
flotation speed, increase the concentrate quality and energy efficiency of the entire
mineral processing process is proposed. In this research, the ultrasound as the main tool
was used.
2 Materials and methods
To facilitate wider adoption of ultrasound technology we use the tools of augmented-
reality to visualize the ultrasound propagation in heterogeneous media by providing a
specific correspondence between the ultrasound data acquired in real-time and a
sufficiently detailed augmented 3D scene. We have established a tablet-based system
for visualizing the propagation of high-energy ultrasound in a heterogeneous medium
in the process of froth flotation using augmented-reality techniques in conjunction with
the streaming data about ultrasound characteristics provided by phased array based on
k-space method. This system gives the operator visual feedback as to the location of the
ultrasonic spot generated by the elements of the phased array, the characteristics of the
ultrasound beam, and look inside the flotation tank.
For programming, the AR tools, interactive objects and creating the markers, the
method of fiber spaces (k-space) for modeling of ultrasonic wave propagation in an
inhomogeneous medium using coarse grids, with maintaining the required accuracy
was used [1; 11; 19].
We describe the ultrasonic waves propagation depending on the mass conservation
equations, momentum conservation law and the equation of state using the first order
dual equations, which can be summarized as follows [1; 5]
p x , t
x c 2 x v x , t x p x , t , (1)
t
v x , t
x p x , t 0 , (2)
t
where p x , t – the time and space dependent ultrasound pressure perturbations (x –
3D Cartesian axis (x, y, z)); x is the spatially dependent density; c x is the spatial
dependent sound speed; v x , t is the velocity of the particle and x is the
absorption coefficient which equivalent to the inverse of the relaxation time.
Let’s represent all absorption effects with one relaxation time. From (2), the
simplified equation can be written as follows
v x , t p x , t
(3)
t x
83
We differentiate (1) with respect to time and variations in (2), and the final equation
can be represented as follows
2 p x , t p x , t
2
x с2 x v x , t x , (4)
t t t
v x , t
p x,t v x , t x 2 p x , t 0 , (5)
t t
Taking into account the permutations (4)
x p x , t 1 2 p x , t
v x , t (6)
t x с 2 x t x с2 x t 2
By substituting this equation in (5), we obtain
x p x , t 2 p x , t 1
2
2 2
p x , t p x 2 p x , t 0 , (7)
с x t с x t x
The simplification of the pressure deviation to the density gradient can be represented
as follows
p x , t 2 p x , t p x , t p x
, (8)
x x x
2
Taking into account (7), eq. (8) can be represented as follows
1 1 2 p x , t x p x , t
p x , t , (9)
x x с 2 x t 2 x с 2 x t
This is a linear wave equation of ultrasonic wave propagation in the heterogeneous
medium with the absorption parameters.
Let’s simplify (9) by separating the parameters of the sound velocity c x and
density x from the second derivatives of pressure taking into account the spatial
and temporal variables to solve the problem of ultrasound propagation using the fiber
space method.
The original equation can be written in the form
1 1 2 p x , t
p x , t 0, (10)
x x с 2 x t 2
The normalized pressure can be represented as follows
84
p x,t
x,t
px
By substituting this equation in (10) we obtain
1
1 1 2 x 2 x , t
p 2 x , t x , t 2
x x с x t 2
After simplifying
1 1 1 2 x , t
2 x , t 2 x x , t 2 2 x
с2 x t 2
Taking into account further simplifications the equation takes the form
2
1 x , t 1 2 12 с02 2 x , t
2 x , t
с02 t 2
с
0
с02
x 2
1
2
x
2
x , t
1
с x t
2
Even more, simplification can be obtained by determining the functions q(r, t) and
v(r, t) efficient sources, which can be summarized as follows
1 1
q x , t с02 2 x x , t 2 2
x
с2
v x , t 2 0 1 x , t
c x,t
By simplifying (11) we obtain
2
1 x,t 1 2v x , t
2 x , t q x , t , (11)
с02 t 2 с02 t 2
This equation can be easily transformed into the frequency domain by using the three-
dimensional spatial Fourier transform as follows
2
1 F k,t 1 2V k,t
k 2 F k,t Q k,t , (12)
c02 t 2 c02 t 2
where F(k, t), Q(k, t) and V(k, t) – three-dimensional spatial Fourier transformation of
values x , t , q x , t and v x , t respectively. Equation (12) satisfies the total
wavefield and is defined as the sum of the incident and scattered field
x , t i x , t s x , t , and the scattered wave field.
85
2
1 x,t
2 x , t 0.
с02 t 2
For the case of an inhomogeneous medium, we introduce an additional source
w x , t s x , t v x , t and by substituting it into (13) we obtain the following
expression
2W k , t
k 2 с02 W k , t V k , t Q k , t , (13)
t 2
с 2 x
where V k , t F 1 2 i x , t w x , t ;
с0
1
Q k , t с02 F x 2 2 x i x , t w x , t v x , t
where F is a spatial Fourier transform.
Let’s use the substandard finite difference approach to solve this equation [10; 15].
Discretization of the time derivative gives
с k t
W k , t t 2W k , t W k , t t 4sin 2 0
2
, (14)
Q k,t
V k , t W k , t 2 2
с0 k
Consider the wave equation on the grayscale for the fiber space method (k-space),
which includes the non-linear characteristic of ultrasound, which can be represented as
follows [10]:
1 1 2 x , t x 2 2 x , t
2 x , t x x , t 2
x с2 x t 2 0 с04 t 2
where 2 x , t is the nonlinearity source, x is the nonlinearity coefficient. The
harmonic oscillations equation can be represented as follows
с k VNL2 k , t W 2 k , t Q k , t ,
2W 2 k , t
t 2
2
0
2
(15)
where w2 x , t s x , t vNL 2 x , t – additional source; W 2 k , t is a spatial
Fourier transform.
86
с2 x 2
vNL 2 x , t 2 0 1 x , t s x , t 2 s x , t i x , t
с x 0 с02
After the spatial Fourier transformation, the equation can be expressed as follows
с2 x 2
2 0 1 i x , t w2 x , t
с x 0 с02
s x , t 2 s x , t i x , t
VNL2 k , t F ;
x 2
2
s x , t 2 s x , t i x , t
0 с0
1
Q k , t F с02 x 2 i x , t w2 x , t vNL 2 x , t ;
x
The introduction of the nonlinearity term in fiber space method makes it easier to
calculate the actual relief temperature in heterogeneous large scale models.
Let’s simulate the ultrasonic pressure field propagation in a heterogeneous medium
using k-Wave toolbox (Matlab) which is designed for time domain ultrasound
simulations in complex media like heterogeneous pulp. The simulation functions of this
software are based on the k-space method and are both fast and easy to use [3; 18].
The net pressure of all piezoelectric elements can be obtained by adding the effects
of each source and written in the form
n
Pnet x, y, z pi x, y, z . (16)
i 1
Due to attenuation, the useful power at the point (x, y, z) is given by [18]
Pnet2 x, y, z
q x, y , z , (17)
c
The total energy at a point (x, y, z) is given by
p 2 x, y , z
I x, y , z , (18)
2 c
where I(x, y, z) – intensity at the point (x, y, z), W/m2.
The results of the ultrasonic wave propagation through a heterogeneous medium
with density ρ = 1500 g/m3, for source strength of 1 MPa and tone burst frequency of
1 MHz for 16-element, phased array with a focus distance of 20 mm are shown on
Fig. 1. The central slice absorption distribution in grayscale as a background and the
square of the pressure distribution on the surface of this background are shown.
87
Fig. 1. Total beam pattern using the maximum of recorded pressure
The final pressure field (a), the maximum pressure (b) and standard pressure (c) of the
beam are shown in Fig. 2. The transducer focus and sidelobes are visible.
Fig. 2. Ultrasonic wave propagation in a heterogeneous medium: a) the final pressure field,
b) the maximum pressure c) the RMS pressure
Fig. 3. The shape of the main wavefront
The linear cross-section of the focus in the x direction is shown in Fig. 4: 1) for the
single source; 2) simulation by a k-space method in the water; 3) in a heterogeneous
88
medium.
Fig. 4. The simulation results comparison of the normalized square of pressure for: 1) a simple
screened source, 2) modeling by the k-space method in a homogeneous medium (water) and
3) in the inhomogeneous medium (pulp) along the axis: a) – z and b) – x.
3 Results
Based on the obtained results, augmented reality tools (interactive objects, markers)
(see Figure 5, 6) were developed in the AR Editor, which are implemented based on
the following algorithm:
STEP 1. Image Capture
o STEP 1.1 Connecting the camera
o STEP 1.2 Capturing video from the camera
o STEP 1.3 Reading video frame frames
STEP 2. Recognition of special points and descriptors
o STEP 2.1 Finding special points on the image
o STEP 2.2 Calculation ─ singular points descriptors
STEP 3. Comparison of descriptors calculated in STEP 2.2 with the database of
marker descriptors. Getting id markers, point vector
STEP 4. Reproduction of relevant content
o STEP 4.1. If the returned number of points in the vector is greater than the
specified value, then go to STEP 4.2. Otherwise, in STEP 4.4.
o STEP 4.2 Getting the content of the marker from the database
o STEP 4.3 Display the contents of the marker on the image
o STEP 4.4 Displaying the image on the screen
The developed algorithm and tools of augmented reality were introduced into the
adaptive control system of the pulp gas phase in the iron ore flotation process using a
control action on the basis of high-energy ultrasound dynamic effects generated by
ultrasonic phased arrays (see Figure 7). The adaptive control system based on the
ultrasonic phased array is placed on the wall of the flotation machine. In accordance
with the above method and the developed algorithm, the system generates a marker in
which the current state of the system is embedded. The tools of augmented reality allow
89
seeing the field of ultrasound propagation, its characteristics, as well as the effect of the
dynamic effects of ultrasound on the change in the gas phase during the flotation
process.
Fig. 5. AR tools (interactive objects, markers) developing in the AR Editor
Fig. 6. AR k-space marker and displaying the contents of the marker on the image
Fig. 7. Augmented reality tools in the adaptive control system of the pulp gas phase of the iron
ore flotation process
90
Conclusions
To build a model of the ultrasonic field in a randomly inhomogeneous medium, the
fiber spaces method (k-space), which increased the accuracy of parameter estimation
field is used. The tools of augmented reality based on k-space methods were developed,
which allow to facilitate wider adoption of ultrasound technology and visualize the
ultrasound propagation in heterogeneous media by providing a specific correspondence
between the ultrasound data acquired in real-time and a sufficiently detailed augmented
3D scene.
References
1. Bergmann, L.: Der Ultraschall: Und Seine Anwendung in Wissenschaft Und Technik. S.
Hirzel Verlag, Stuttgart (1954)
2. Hruntova, T.V., Yechkalo, Yu.V., Striuk, A.M., Pikilnyak, A.V.: Augmented Reality Tools
in Physics Training at Higher Technical Educational Institutions. In: Kiv, A.E., Soloviev,
V.N. (eds.) Proceedings of the 1st International Workshop on Augmented Reality in
Education (AREdu 2018), Kryvyi Rih, Ukraine, October 2, 2018. CEUR Workshop
Proceedings 2257, 33–40. http://ceur-ws.org/Vol-2257/paper04.pdf (2018). Accessed 30
Nov 2018
3. Introduction to Phased Array Ultrasonic Technology Applications: Olympus Guideline.
Olympus, Waltham (2017)
4. Lvov, M.S., Popova, H.V.: Simulation technologies of virtual reality usage
in the training of future ship navigators. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings
of the 2nd International Workshop on Augmented Reality in Education (AREdu 2019),
Kryvyi Rih, Ukraine, March 22, 2019, CEUR-WS.org, online (2020, in press)
5. Mast, T.D., Souriau, L.P., Liu, D.-L.D., Tabei, M., Nachman, A.I., Waag, R.C.: A k-Space
Method for Large-Scale Models of Wave Propagation in Tissue. IEEE Transactions on
Ultrasonics, Ferroelectrics, and Frequency Control 48(2), 341–354 (2001).
doi:10.1109/58.911717
6. Meisner, J., Donnelly, W.P., Roosen, R.: Augmented reality technology. US Patent
6,625,299 B1, 23 Sept 2003
7. Mintii, I.S., Soloviev, V.N.: Augmented Reality: Ukrainian Present Business and Future
Education. In: Kiv, A.E., Soloviev, V.N. (eds.) Proceedings of the 1st International
Workshop on Augmented Reality in Education (AREdu 2018), Kryvyi Rih, Ukraine,
October 2, 2018. CEUR Workshop Proceedings 2257, 227–231. http://ceur-ws.org/Vol-
2257/paper22.pdf (2018). Accessed 30 Nov 2018
8. Modlo, Ye.O., Semerikov, S.O., Bondarevskyi, S.L., Tolmachev, S.T., Markova, O.M.,
Nechypurenko, P.P.: Methods of using mobile Internet devices in the formation of the
general scientific component of bachelor in electromechanics competency in modeling of
technical objects. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings of the 2nd International
Workshop on Augmented Reality in Education (AREdu 2019), Kryvyi Rih, Ukraine, March
22, 2019, CEUR-WS.org, online (2020, in press)
9. Modlo, Ye.O., Semerikov, S.O., Nechypurenko, P.P., Bondarevskyi, S.L., Bondarevska,
O.M., Tolmachev, S.T.: The use of mobile Internet devices in the formation of ICT
component of bachelors in electromechanics competency in modeling of technical objects.
In: Kiv, A.E., Soloviev, V.N. (eds.) Proceedings of the 6th Workshop on Cloud
91
Technologies in Education (CTE 2018), Kryvyi Rih, Ukraine, December 21, 2018. CEUR
Workshop Proceedings 2433, 413–428. http://ceur-ws.org/Vol-2433/paper28.pdf (2019).
Accessed 10 Sep 2019
10. Morkun, V., Morkun, N., Pikilnyak, A.: Modeling of ultrasonic waves propagation in
inhomogeneous medium using fibered spaces method (k-space). Metallurgical and Mining
Industry 2, 43–48 (2014)
11. Morkun, V., Morkun, N., Pikilnyak, A.: The adaptive control for intensity of ultrasonic
influence on iron ore pulp. Metallurgical and Mining Industry 6, 8-11 (2014)
12. Morkun, V., Morkun, N., Pikilnyak, A.: Ultrasonic phased array parameters determination
for the gas bubble size distribution control formation in the iron ore flotation. Metallurgical
and Mining Industry 3, 28–31 (2014)
13. Morkun, V., Morkun, N., Pikilnyak, A.: Ultrasonic testing of pulp solid phase concentration
and particle size distribution considering dispersion and dissipation influence. Metallurgical
and Mining Industry 1, 9–13 (2015)
14. Pikilnyak, A.: Adaptive control system of the iron ore flotation using a control action based
on high-energy ultrasound. Metallurgical and Mining Industry 2, 27–30 (2015)
15. Pikilnyak, A.: The gas bubble size parameters monitoring and control method.
Metallurgical and Mining Industry 7, 19-21 (2015)
16. Striuk, M.I., Semerikov, S.O., Striuk, A.M.: Mobility: a systems approach. Information
Technologies and Learning Tools 49(5), 37–70 (2015). doi:10.33407/itlt.v49i5.1263
17. Syrovatskyi, O.V., Semerikov, S.O., Modlo, Ye.O., Yechkalo, Yu.V., Zelinska, S.O.:
Augmented reality software design for educational purposes. In: Kiv, A.E., Semerikov,
S.O., Soloviev, V.N., Striuk, A.M. (eds.) Proceedings of the 1st Student Workshop on
Computer Science & Software Engineering (CS&SE@SW 2018), Kryvyi Rih, Ukraine,
November 30, 2018. CEUR Workshop Proceedings 2292, 193–225. http://ceur-ws.org/Vol-
2292/paper20.pdf (2018). Accessed 21 Mar 2019
18. Tabei, M., Mast, T.D., Waag, R.C.: A k-space method for coupled first-order acoustic
propagation equations. Journal of the Acoustical Society of America 111(1), 53–63 (2002).
doi:10.1121/1.1421344
19. Ul'trazvuk: Malaja jenciklopedija (Ultrasound: Small Encyclopedia). Soviet Encyclopedia,
Moscow (1979)
20. Zelinska, S.O., Azaryan, A.A., Azaryan, V.A.: Investigation of Opportunities of the
Practical Application of the Augmented Reality Technologies in the Information and
Educative Environment for Mining Engineers Training in the Higher Education
Establishment. In: Kiv, A.E., Soloviev, V.N. (eds.) Proceedings of the 1st International
Workshop on Augmented Reality in Education (AREdu 2018), Kryvyi Rih, Ukraine,
October 2, 2018. CEUR Workshop Proceedings 2257, 204–214. http://ceur-ws.org/Vol-
2257/paper20.pdf (2018). Accessed 30 Nov 2018
21. Zinonos, N.O., Vihrova, E.V., Pikilnyak, A.V.: Prospects of Using the Augmented Reality
for Training Foreign Students at the Preparatory Departments of Universities in Ukraine.
In: Kiv, A.E., Soloviev, V.N. (eds.) Proceedings of the 1st International Workshop on
Augmented Reality in Education (AREdu 2018), Kryvyi Rih, Ukraine, October 2, 2018.
CEUR Workshop Proceedings 2257, 87–92. http://ceur-ws.org/Vol-2257/paper10.pdf
(2018). Accessed 30 Nov 2018