<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>3D Radio Holographic Images Synthesis and Filtration on Multiprocessor Computing Systems</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Alexey A. Kalmykov</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Vadim A. Dobryak</string-name>
          <email>dobryak1958@gmail.com</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Andrey A. Kalmykov</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Anton S. Kurilenko</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Elena N. Akimova</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Aliya F. Skurydina</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Vladimir E. Misilov</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Krasovskii Institute of Mathematics and Mechanics, Ural Branch of RAS</institution>
          ,
          <addr-line>Yekaterinburg</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Ural Federal University</institution>
          ,
          <addr-line>Yekaterinburg</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
      </contrib-group>
      <fpage>36</fpage>
      <lpage>44</lpage>
      <abstract>
        <p>The work describes an algorithm for 3D radio holographic images synthesis in a subsurface sounding system designed by the authors. The algorithm was parallelized and implemented for multicore processors and NVIDIA GPU. Computational complexity of the algorithm was estimated. Numerical experiments were performed for estimating the algorithm performance and e ciency. Experiments were carried out on multicore processors with vector instructions set and NVIDIA GPU. The developed system can locate inhomogeneity (size of which can be a few centimeters) in doublelayered reinforced concrete at the depth of a few tens of centimeters.</p>
      </abstract>
      <kwd-group>
        <kwd>3D images</kwd>
        <kwd>subsurface radiolocation</kwd>
        <kwd>fast Fourier transform</kwd>
        <kwd>parallel computations</kwd>
        <kwd>linear ltration</kwd>
        <kwd>CUDA</kwd>
        <kwd>OpenMP</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        A number of problems, which are solved using the subsurface sounding [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], is
growing today. It is stipulated by increased availability of locators quality, as well
as, by increased performance of modern computational systems. Application of
the parallel or quasi-parallel systems allows one to perform the sounding and
image synthesis in a fraction of a second. The wide range of problems includes
high-performance passenger screening systems, searching for the hidden objects
in buildings, computer vision systems, archeology, etc. The 3D radio holographic
subsurface locator developed by authors is described in [2{6]. The basic
operating principles include: application of the sounding signal with a linear frequency
modulation, correlation lter based on processing the re ected signal, an
internal coherence of the system, and radio holographic synthesis of 3D images. All
these principles allow the system to obtain high spatial resolution by using
ultrawideband sounding signal, high angular resolution by using aperture synthesis
with portable antennas, and MIMO-lines or matrices. Further modi cation of
the image synthesis algorithm considering refraction was developed. Now for
2
      </p>
      <p>
        Image synthesis algorithm
The algorithm comprises the following steps [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]:
      </p>
      <p>1. There is a set of beat-frequency waveforms fsk(t)g; k = 1; K, for di erent
positions f(xk; yk; zk)g of antenna.</p>
      <p>
        2. Compute the complex spectrums of these waveforms using the Fourier
transform
practical application of the system, it is essential to use e ective software and
hardware for fast processing of radio signals and showing results in the real time.
The authors see two ways of using hardware to implement fast signal
processing. The rst one is to use specialized computers with signal processors and
programmable logic devices. The second one is to use general-purpose personal
computers with high-performance coprocessors. Combination of these two ways
can be used [
        <xref ref-type="bibr" rid="ref7 ref8">7, 8</xref>
        ]. In this paper, we estimate performance of one parallel
algorithm for 3D image synthesis using the CPU and GPU, estimate the resolution
obtained, and propose the way to its increasing.
(2)
(3)
Sk(f ) = F fsk(t)g:
(1)
The delay is = f Tm= f , so, we get a set fSk( )g. Here, Tm is the modulation
period and f is the frequency deviation. The algorithmic complexity of this
step is O(N log(N ) K), where N is the number of quantization steps for one
signal.
      </p>
      <p>3. Perform phasing for the whole set</p>
      <p>Sk0( ) = Sk( ) e j 0( );
I(x; y; z) = k X Sk0( )k;</p>
      <p>K
where 0( ) = 2 (f0 f =2) . Complexity of this step is O(N K).</p>
      <p>4. For all coordinates of the required volume (x; y; z), calculate the total
intensity
where k is the total delay of the echo from the (x; y; z) point considering possible
refractions at the media interfaces. The complexity of this step is O(M K), where
M = X Y Z is the total number of voxels in the synthesized image, X; Y; Z
are the voxel resolutions for each coordinate.</p>
      <p>The total algorithmic complexity of the algorithm is O(K(N log(N ) + M )).
To achieve the desired resolution of synthesized images, we have to impose
certain restrictions onto all components of this expression. Application of the system
with the serial sounding allows one to use parallelization for each single signal
processing (on each step of the Fourier transform computation and
accumulation) and to nd values independently for di erent points using T threads. Thus,
the complexity can be reduced to O( KT (N log(N ) + M )).</p>
      <p>Parallelization and numerical experiments results
Application of the multithreaded system allows one to process the data for each
thread independently. So, application of T threads reduces time complexity by T
times. However, this also increases space complexity by requirement to store copy
of the data for each thread. Combination of these methods can be used. In the
experiment, the synthesized image has the following parameters: the resolution
in voxels is 100 100 300, the beat-frequency waveforms set K has the length
of 2048, and the number of quantization steps is 216. Both the serial and parallel
methods were applied to the same image. Table 1 shows the computation times
for the serial method and parallel one with four threads. It is seen that the
computation time increases linearly with the number of signals.</p>
      <p>Another way to increase the implementation performance is to use
optimization for the certain processor architecture. For example, contemporary Intel
processors support the vector instructions sets. This allows one to increase the
performance of oating-point operations signi cantly.</p>
      <p>The results of the automatically vectorized program for the Intel Core i7
processor using the Intel C++ Compiler 14.0 are shown in Table 1. The thread
parallelization was performed using the OpenMP technology. The number of
signals is K = 2048.</p>
      <p>The algorithm was also implemented for GPU using the NVIDIA CUDA
technology. The experiments were carried out using the NVIDIA GTX 780Ti
(2880 cores, 875 MHz). Table 3 shows that this implementation allows one to
produce an image with 15 frames per second.</p>
      <p>The experiments were also performed using a PC with the Intel Core i7
3610M (2.4 GHz, 9 logical cores) processor with 16 GB of RAM. The algorithm
parameters were: M = 106 and N = 262144. The time shown in tables is the
minimal time of 1000 experiments with the same conditions. The FFT algorithm
was taken from the FFTW and NVIDIA cuFFT libraries.
The problem of the low-sized subsurface object visualization motivates creating
devices with characteristics extremely close to their theoretical physical limits.
Some of the fundamental bases of those limits are a wavelength of the probing
signal, aperture sizes, and signal losses in the probed mediums. At the same
time, methods of additional image ltering facilitate reaching these limits, which
is useful in solving various applied problems.</p>
      <p>For example, consider a problem of probing di erent building structures.
The purpose of linear ltering consists of highlighting the low-sized objects on a
background of high-intensity re ections from reinforcements and wall surfaces.
Would the ltering method based on the minimal prior information (e.g. that
the wall is considered to be at, the reinforcement bars to be long, etc.) be still
e ective?</p>
      <p>The principle of the lter operation is the following. Figure 1 (top) shows the
simulated two-dimensional image of the reinforcement lattice and the absolute
value of its spatial frequency spectrum.</p>
      <p>It can be shown that the absolute value does not depend on the shift of
the lattice with respect to the picture frame. The shift occurs only in the
phasefrequency spectrum. The uctuations of the absolute values of frequency
components depend on the lattice's step size. The signi cant volume of spatial harmonic
curves for a rectangular reinforcement lattice is concentrated along the
coordinate axes. With rotation of the lattice, the spectrum rotates about the vertical
axis.</p>
      <p>Horizontally oriented ribs correspond to the harmonics located alongside one
of the axes, and the vertically oriented ribs correspond to the harmonics located
alongside the other one. Extending this conclusion to the third dimension, we
can be assured that any orthogonal plain in the 3D image corresponds to the
spectrum concentrated alongside a single plain as well.</p>
      <p>This fact suggests using a linear 3D lter with the spatial-frequency spectrum
that equals 0 for any coordinates that have at least one component at zero and
equals 1 in any other case. The lter is supposed to be e ective in clearing any
plains and lines from the images that are orthogonal or parallel to any of the
coordinate axes. However, the rate of distortion of intensity level and the form
of low-sized objects remain vital.</p>
      <p>Figure 1 (bottom) shows the simulated image of a low-sized object and its
spectrum. As we can see, the signi cant part of spectrum of the object's image is
spread along all the frequencies. This fact proves that our lter can be e ective.</p>
      <p>Figure 2 shows summary images and their spectra before (top) and after
(bottom) ltering. The object's intensity is slightly reduces and the cross-like
beams appeared, but these are admissible distortions.</p>
      <p>The algorithm for the lter is implemented in the frequency domain by using
forward and backward Fourier transform with transitional multiplication of the
spectrum by the spatial-frequency function (see Fig. 3). The complexity of the
Fourier transform is O(M 3 log(M )), the complexity of multiplication is O(M 3).
Thus, the algorithmic complexity of ltration is O(M 3).
Analysis of algorithmic complexity of the algorithm suggested for synthesis of 3D
radio holographic images was carried out. Application of the serial and parallel
processing was considered. The experiments were carried out for performance
increasing by application of the parallel computing and vector instructions set.
It was shown that by using the GPUs, it is possible to achieve performance
su cient for the real-time synthesis of the images. We have shown possibility of
using the secondary linear ltering for the radio-frequency 3D images to reduce
the masking e ect of wall surfaces and reinforcement lattice as well as highlight
the low-sized masked objects. The developed system can locate inhomogeneity
(size of which can be a few centimeters) in double-layered reinforced concrete at
the depth of a few tens of centimeters.</p>
      <p>Acknowledgment This work was partially supported by Russian Foundation
for Basic Research (Project No. 15-01-00629).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Grinyov</surname>
            ,
            <given-names>A. U.</given-names>
          </string-name>
          :
          <article-title>Questions of subsurface radiolocation (in Russian)</article-title>
          .
          <source>Radiotechnika</source>
          , Moscow (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Dobryak</surname>
            ,
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            , Al.
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            ,
            <given-names>An. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kurilenko</surname>
            ,
            <given-names>A. S.:</given-names>
          </string-name>
          <article-title>Georadar with 3D images synthesis (in Russian)</article-title>
          .
          <source>In: Proceedings of XI international scienti c and technical conference Physics</source>
          and
          <article-title>technical applications of wave processes</article-title>
          .
          <source>UrFU</source>
          ,
          <string-name>
            <surname>Yekaterinburg</surname>
          </string-name>
          (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Dobryak</surname>
            ,
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            , Al.
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            ,
            <given-names>An. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kurilenko</surname>
            ,
            <given-names>A. S.</given-names>
          </string-name>
          :
          <article-title>Theory and practice of three-dimensional radio frequency visualization of objects (in Russian)</article-title>
          .
          <source>In: The 23rd Int. Crimean Conf. Microwave &amp; Telecommunication Technology (CriMiCo2013)</source>
          , pp.
          <fpage>11691170</fpage>
          .
          <string-name>
            <surname>Sevastopol</surname>
          </string-name>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Dobryak</surname>
            ,
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            , Al.
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            ,
            <given-names>An. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kurilenko</surname>
            ,
            <given-names>A. S.:</given-names>
          </string-name>
          <article-title>Application of MIMO-lines in the problem of three-dimensional radio frequency visualizations of subsurface objects (in Russian)</article-title>
          .
          <source>In: The 23rd Int. Crimean Conf. Microwave &amp; Telecommunication Technology (CriMiCo2013)</source>
          , pp.
          <fpage>11951196</fpage>
          .
          <string-name>
            <surname>Sevastopol</surname>
          </string-name>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Dobryak</surname>
            ,
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            , Al.
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            ,
            <given-names>An. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kurilenko</surname>
            ,
            <given-names>A. S.:</given-names>
          </string-name>
          <article-title>Additional focusing in the problem of three-dimensional radio frequency visualizations of subsurface objects (in Russian)</article-title>
          .
          <source>In: The 23rd Int. Crimean Conf. Microwave &amp; Telecommunication Technology (CriMiCo2013)</source>
          , pp.
          <fpage>11851186</fpage>
          .
          <string-name>
            <surname>Sevastopol</surname>
          </string-name>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Kalmykov</surname>
            , Al.
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dobryak</surname>
            ,
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kalmykov</surname>
            ,
            <given-names>An. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kurilenko</surname>
            ,
            <given-names>A. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Akimova</surname>
            ,
            <given-names>E. N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Skurydina</surname>
            ,
            <given-names>A. F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Misilov</surname>
            ,
            <given-names>V. E.</given-names>
          </string-name>
          :
          <article-title>Parallel algorithms for synthesizing and processing of 3D radio holographic images (in Russian)</article-title>
          .
          <source>In: Proceedings of the 10th Annual International Scienti c Conference on Parallel Computing Technologies</source>
          , pp.
          <fpage>521529</fpage>
          , CEUR Workshop Proceedings, Aachen (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Barkhatov</surname>
            ,
            <given-names>A. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kozlov</surname>
            ,
            <given-names>A. S.:</given-names>
          </string-name>
          <article-title>Fast calculation of time-frequency function in radar using GPUs (in Russian)</article-title>
          .
          <source>Radiolokatsiya i radionavigatsiya. 5</source>
          ,
          <issue>4246</issue>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Ampilov</surname>
            ,
            <given-names>O. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pyatkin</surname>
            ,
            <given-names>A. K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Topchiev</surname>
            ,
            <given-names>S. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nikitin</surname>
            ,
            <given-names>M. V.</given-names>
          </string-name>
          :
          <article-title>Digital signal processing in coherent monopulse radar (in Russian)</article-title>
          .
          <source>OAO Radiophysica</source>
          , Moscow (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>