=Paper=
{{Paper
|id=Vol-2391/paper21
|storemode=property
|title=Method for reconstructing the real coordinates of an object from its plane image
|pdfUrl=https://ceur-ws.org/Vol-2391/paper21.pdf
|volume=Vol-2391
|authors=Vladimir Nesterov,Vasily Mukhin,Dmitry Nesterov
}}
==Method for reconstructing the real coordinates of an object from its plane image ==
https://ceur-ws.org/Vol-2391/paper21.pdf
Method for reconstructing the real coordinates of an object
from its plane image
V N Nesterov1,2, V M Mukhin1 and D V Nesterov1
1
JSC «Samara Electromechanical Plant», Stepan Razin street, 16, Samara, Russia, 443099
2
Samara National Research University, Moskovskoe shosse, 34А, Samara, Russia, 443086
e-mail: nesterov.ntc@gmail.com
Abstract. Original method of reconstructing the real coordinates of moving objects from their
plane images is presented. The method uses multidimensional test objects whose parameters
are measures and which are known with high accuracy. In order to ensure the process of
measuring movements in real space a test object must be connected with a real object and
mathematical models of images of multicomponent movements of a test object must be
formed. The parameters of test objects that are used in the named mathematical models are
vectors. Such models are used in the construction of systems of measurement equations the
solution of which gives the desired components movements of a moving object in 3D space.
The method was experimentally tested on specially created stand.
1. Introduction
Problem of the impossibility of recovering the real coordinates of objects in three-dimensional space
from their plane image is one of the most serious problems in robotics. This problem was discussed in
the fundamental work of well-known experts in the field of robotics and technical vision K. Fu, R.
González and K. Li [1]. A more modern work provides a broad overview of various approaches that
are used to solve this ill-posed problem [2]. Binocular vision is traditionally used to solve this problem
in robotics and other areas. However, this solution leads to additional problems concerning the
calibration of cameras, their placement in space, special algorithms for processing the received video
information, etc. If you look at the problem more broadly, you should mention the works related to
geo-informatics [3] or architecture [4]. Currently, geographic information systems are increasingly
complemented by tools for working with spatial geometric objects, which allows solving problems of
3D modeling of objects that are spatially referenced to the surface of the Earth. The reconstruction of
3D coordinates of objects from plane projections in photogrammetry is solved by processing stereo
pairs of images [5]. To illustrate the problem consider the process of imaging a 3D object on the
surface of a video camera image. With some simplifications the imaging circuit is shown on the figure
1.
On the figure 1 the image plane coincides with the plane x, y of the video camera's coordinate
system {o, x, y, z} , and the optical axis, which is set in the center of the lens, is directed along the axis z
The center of the image plane is the origin, and the center of the lens has coordinates (0, 0, λ) .
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019)
Image Processing and Earth Remote Sensing
V N Nesterov, V M Mukhin and D V Nesterov
Denote the coordinates of a point in 3D space located at a distance in front of the lens, as ( X , Y , Z ).
For all points of controlled space: Z > λ . From consideration of the corresponding right-angled
triangles on the figure 1, we can write:
x0 X1 y0 Y
= , = 1 . (1)
λ λ − Z1 λ λ − Z1
Then the relationship between the coordinates of points M 1 ( X 1 , Y1 , Z 1 ) and m( x0 , y0 ) is
determined by the functions:
λX 1 λY1
x0 = , y0 = . (2)
λ − Z1 λ − Z1
They can be used to go from the Cartesian coordinates of a point M 1 ( X 1 , Y1 , Z 1 ) to the
coordinates of a point of its image m( x0 , y0 ) .
Figure 1. Image formation scheme.
Suppose that the point specified in the image has coordinates ( x 0 , y 0 , 0 ) . In this case, the entire
image plane on the z axis has a coordinate z = 0 . The point of the image m( x0 , y0 ) corresponds to all
points lying on a line that passes through the points with coordinates ( x 0 , y 0 , 0 ) and (0, 0, λ ) . The
equations of this line in Cartesian coordinates are:
x y
X 1 = 0 (λ − Z 1 ) , Y1 = 0 (λ − Z1 ) . (3)
λ λ
The ambiguity of this transformation is obvious. It is impossible to uniquely reconstruct a point
M 1 ( X 1 , Y1 , Z 1 ) in 3D space by the coordinates of its image m( x0 , y0 ) .
In all known methods for solving this ill-posed problem, including in the above-mentioned works,
information redundancy is organized, for example, by using an additional camera (binocular vision) or
using another method of obtaining at least two projections of a real three-dimensional object.
To solve this problem in recent years, the authors develop an original scientific direction, called:
the method of multidimensional test objects in optical measurement systems [6].
To ensure the measurement process, a test object must be connected with a real object. And test
object must have known geometrical parameters that are used as measures [7, 8]. At the same time,
mathematical models of images of multicomponent movements are formed, which contain the
parameters of test objects. These parameters are included in the models in the form of vectors and they
are called tests. This is the basis of the method. Next we look at the basics of this method in more
detail.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 154
Image Processing and Earth Remote Sensing
V N Nesterov, V M Mukhin and D V Nesterov
2. The main points of the multidimensional test objects method
Note that the following provisions and models have a specific practical orientation and, above all,
relate to the restoration of informative movements of 3D objects in space. Therefore, it should be
borne in mind certain restrictions on the subject area, although according to the authors'
considerations, the provisions of the method can be extended to a wider range of tasks.
Since we are talking about the reconstruction of the components of the multidimensional
movement of objects in three-dimensional space, which are vector quantities, the elements of
multidimensional test objects included in the models are also vector quantities. A feature of the
method is that the parameters of a multidimensional test object reflect the multidimensionality of will
be measured movements and are functionally associated with them in the models in the process of
forming the corresponding measurement and computational algorithms:
X x (r,τ ) = F (x1x (r,τ ),...,x px (r,τ ), L1x ,...,L qx ) ;
X y (r,τ ) = F(x1 y (r,τ ),..., x py (r,τ ), L1 y ,..., L qy ); (4)
X z (r,τ ) = F(x1z (r,τ ),..., x pz (r,τ ), L1z ,..., L qz ),
where X x (r,τ ), X y (r,τ ), X z (r,τ ) are projections of multi-component movements on the axis of the
Cartesian coordinate system; r is radius-vector from the beginning of the base coordinate system to
the controlled point of the object under study; τ is time; F is components link function
x1k (r,τ ) ,..., x pk (r,τ ) and L1k ,..., L qk L1k ,..., L qk of the coordinate component L k ( k ∈ {x, y, z} )
of the multidimensional test L ; x1k (r,τ ) ,..., x pk (r,τ ) are informative components of the k
coordinate component of the multicomponent movement X(r,τ ) ; p is number of informative
components of multicomponent movement; L1k ,..., L qk are components of the k coordinate
component L k of the multidimensional test L ; p is number of components of the k coordinate
component L k of the multidimensional test L .
As already noted, the principal feature of the model (4) is the introduction of test objects
L1k , ..., L qk , which must be indicated in vector form as informative components known in it.
Figure 2 shows, as an illustration, a two-dimensional test object, which was obtained by a
combination of two one-dimensional.
The test object is located in the plane О0X 0Y0 and has the following parameters (tests), which are
known with high accuracy:
AO i = nL ABx и BOi = (1 − n ) L ABx , (n = 0,5) ;
CO i = nLCDy и DOi = (1 − n ) LCDy , (n = 0,5) ;
EB = (1 − n ) L ABx и FD = (1 − n ) LCDy , (n = 0,75) .
The test object О 0 X 0 Y0 is a two-dimensional and multicomponent, and its parameters in models (5)
are used in a vector form, for example:
AO i = nL ABx = nL ABx ⋅ i , (n = 0,5) ; BOi = (1 − n ) L ABx = (1 − n ) L ABx ⋅ i , (n = 0,5) ;
CO i = nL CDy = nLCDy ⋅ j , (n = 0,5) ; DOi = (1 − n ) L CDy = (1 − n ) LCDy ⋅ j , (n = 0,5) ;
EB = (1 − n ) L ABx = (1 − n ) L ABx ⋅ i , (n = 0,75) ; FD = (1 − n ) L CDy = (1 − n ) LCDy ⋅ j , (n = 0,75) ,
where i and j are basis vectors whose direction coincides with the direction of the axes О 0 X 0 and
О 0 Y0 .
Here the problem is not set to classify or demonstrate the variety of test objects. Here are
formulated only the basic requirements for them, which are fundamental.
Let us draw an analogy between the parameters (components) of a multidimensional test object and
the components of complex displacements. Then the components of multidimensional tests or their
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 155
Image Processing and Earth Remote Sensing
V N Nesterov, V M Mukhin and D V Nesterov
projections on the coordinate axes will be considered as multicomponent quantities – multicomponent
tests, the components of which are also vector quantities.
Figure 2. 2D test object ABCD.
Accordingly, the general method of forming multidimensional tests and the function of the
connection of their components with multicomponent quantities fall under the basic principles of the
concept of vector multicomponent physical quantities [5] and are formulated as follows:
- multidimensional multicomponent tests are considered as functions of the set of their
components;
- functions of communication of the named components in models of multicomponent tests are
determined by the laws of vector algebra;
- models of vector multidimensional multicomponent tests allow the multivariate representation of
these components, depending on the problem being solved.
- models of vector multidimensional multicomponent tests allow to give alternative representation
of these components, depending on the problem being solved.
Based on the provisions about the multidimensional test object, we define the type of function of
communication F of informative components x1k (r,τ ) ,..., x pk (r,τ ) and components L1k , ..., L qk of
the coordinate component L k of the multidimensional test L in model (4):
{x,y.z} q {x,y,z} p
Fik {x1k (r,τ ), ..., x pk (r,τ ), L1k , ..., L qk } = ∑ ∑ viuk L iuk + ∑ ∑ ηijk x ijk (r,τ ) , (5)
k u =1 k j =1
where i is the sequence number of the communication function; k ∈ { x, y, z} is the set of coordinate
components; u is the serial number of the components of the multi-component test L iuk ; j is the
sequence number of the informative components of the coordinate component of the multicomponent
displacement X k (r, τ ) ; viuk ∈ [0,1] are the weights reflecting the absence – 0 – or the presence –
(0,1] – the corresponding component of a multicomponent test L iuk in model (5); ηijk ∈ [0,1] are the
weights reflecting the absence – 0 – or the presence – (0,1] – the corresponding informative
component x ijk (r,τ ) in model (5).
Using (5) it is possible to present model (4) in the following form:
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 156
Image Processing and Earth Remote Sensing
V N Nesterov, V M Mukhin and D V Nesterov
q p
X ix (r,τ ) = ∑ viux L iux + ∑ ηijx x ijx (r,τ ) ;
u =1 j =1
q p
X iy (r,τ ) = ∑ viuy L iuy + ∑ ηijy x ijy (r, τ ) ; (6)
u =1
q
j =1
p
X iz (r,τ ) = ∑ viuz L iuz + ∑ ηijz x ijz (r,τ ) ,
u =1 j =1
where the vectors L iux , L iuy , L iuz , x ijx , x ijy , x ijz are defined in one-dimensional spaces that
coincide with the corresponding axes of the Cartesian coordinate system.
Such a representation of model (5), while retaining its universal character, provides a mechanism
for adaptation to specific practical tasks through the use of a combination of coefficients viux ∈ [0, 1] ,
viuy ∈ [0,1] , viuz ∈ [0,1] , ηijx ∈ [0,1] , ηijy ∈ [0,1] , ηijz ∈ [0,1] in the field of their definition.
The following is figure 3, which illustrates the process of moving the test object, on the basis of
which the model (6) was obtained. It shows the movement in space of a cruciform shape, which is a
test object, and the reflection of this movement on a flat matrix of the optical instrument.
Figure 3. Illustration for model (6).
Any movement, including multicomponent, is described in accordance with the laws and
provisions of vector algebra. When considering the projections of vector quantities onto the plane and
introducing special agreements, it is possible to significantly simplify the process of synthesizing
complex mathematical models by formally generating them, which is necessary to search for
appropriate measurement and computational algorithms [9]. The information redundancy embodied in
the models in the form of parameters of a multidimensional test object allows us to approach its
solution.
3. Physical realizability of the measurement method
Based on the formulated provisions on multidimensional test objects, we list the method-forming
features, the implementation of which will solve the problem of the physical realizability of the
method:
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 157
Image Processing and Earth Remote Sensing
V N Nesterov, V M Mukhin and D V Nesterov
1. The presence (the ability to form) a system of n equations that are asymmetric with respect to
informative components x1k (r, τ), ..., x pk (r, τ) ( k ∈ { x, y, z} is the set of coordinate components)
displacements of the corresponding image points of the test object:
Y1 (r, τ) = Ψ1 {F1 {x1k (r, τ), ..., x pk (r, τ), L1k , ..., L qk }} ;
(n ≥ p ≥ 2 ) ,
.............................................. (7)
..............................................
Y n (r, τ) = Ψ n {F p {x1k (r, τ), ..., x pk (r, τ), L1k , ..., L qk }} ,
F1 {x1k (r, τ), ..., x pk (r, τ), L1k , ..., Lqk } ≠ ... ≠ F p {x1k (r, τ), ..., x pk (r, τ), L1k , ..., Lqk } , (8)
where Y1 (r, τ ), ..., Yn (r, τ ) are functions of displacement of the corresponding points of the image of
the object being monitored relative to the selected on the image of the points of reference;
F1 {x1k (r, τ), ..., x pk (r, τ), L1k , ..., L qk },..., F p {x1k (r, τ),..., x pk (r, τ), L1k ,..., L qk } are vector functions of the set
of informative components x1k (r, τ ) ,..., x pk (r, τ ) and components L1k , ..., L qk of the coordinate
component L k of a multidimensional test object (of the multidimensional test) L .
2. The ability to realization special measuring and computing algorithms:
x1k (r, τ) = f1 {Y1 (r, τ), ..., Yn (r, τ)} ;
...........................................
........................................... (9)
x pk (r, τ ) = f p {Y1 (r, τ ), ..., Yn (r, τ )} ,
condition of the existence of which, with continuity and differentiability Y1 (r, τ ), ..., Yn (r, τ ) in the
whole range of measurement, is the inequality of zero Jacobian:
∂ Yi (r, τ )
det ≠ 0 i = 1, n , j = 1, p . (10)
∂ x jk (r , τ )
The condition (10) is ensured by the implementation of the “asymmetry” of the values
Y1 (r, τ), ..., Yn (r, τ) relative to their constituent components x1k (r, τ ) ,..., x pk (r, τ ) and
L1k , ..., L qk , which is expressed by the inequality (8).
Obviously, when using a single-channel optical system, the functions ψ1 , ..., ψ n are the same. If
we use the transmission coefficient σ of the optical converter, then the system of equations (7) can be
rewritten as follows:
Y1 (r, τ) = σ{F1 {x1k (r, τ), ..., x pk (r, τ), L1k , ..., L qk }} ;
(n ≥ p ≥ 2 ) .
.............................................. (11)
..............................................
Yn (r, τ) = σ{F p {x1k (r, τ), ..., x pk (r, τ), L1k , ..., L qk }} ,
Having considered the system (11) we can understand all the perspectives of the optical
measurement method. For a more detailed consideration of the implementation of the proposed
method, refer to [6-9]. The originality and novelty of the measurement method is confirmed by patents
of the Russian Federation for inventions [10–12]. The method was experimentally tested on specially
created stand where the cruciform figure was used as a test object. The details of the experiment and
its results are presented in a separate section in paper [6].
4. Conclusions
Summing up, we can conclude that, on the basis of the method of multidimensional test objects, the
incorrect problem of reconstructing the coordinates of objects moving in three-dimensional space from
their flat image has been solved.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 158
Image Processing and Earth Remote Sensing
V N Nesterov, V M Mukhin and D V Nesterov
The basics of optical measurements of informative components of multicomponent displacements
based on the method of multidimensional test objects, providing information redundancy at the input
of an optical image receiver, are also presented. Additional information at the system input in the form
of known parameters of multidimensional test objects creates opportunities for improving the quality
and quantity of the resulting information.
Separate attention is required for the problem of generating multidimensional test objects, to
investigate the effects of their type, quantity and optimization of test components (parameters of a
multidimensional test object) on the quality of the measuring system both in the general theoretical
aspect and for solving specific applied problems. These questions should be the subject of further
research.
5. References
[1] Fu K, Gonzalez R and Lee K 1989 Robotics (Moscow: World) p 624
[2] Oswald M R, Toppe E, Nieuwenhuis C and Cremers D 2013 A review of geometry recovery
from a single image focusing on curved object reconstruction Innovations for shape analysis:
models and algorithms (Berlin Heidelberg: Springer-Verlag) 343-378 DOI: 10.1007/978-3-642-
34141-0
[3] Kostyuk Yu L, Paramonov A S and Gritsenko V G 1998 The technology of creating 3D models
of objects on flat projections and its application in geoinformatics Geoinformatics: Theory and
Practice (Tomsk: Tomsk University) 1 96-106
[4] Dzwierzynska J 2016 Reconstructing architectural environment from the perspective image
World multidisciplinary civil engineering-architecture-urban planning symposium, WMCAUS
(ScienceDirect: Proc. engineering) 161 1445-1451
[5] Kostyuk Yu L 2002 Efficient algorithms for processing and displaying graphic data and their
implementation in software packages Abstract of thesis ... doc. tech. of science (Tomsk: Tomsk
University) p 39
[6] Nesterov V N, Mukhin V M and Meshchanov A V 2013 The method of multidimensional test
objects in optical measurement systems (Samara: Publishing House of the Samara Scientific
Centre of the Russian Academy of Sciences) p 224
[7] Nesterov V N and Meshchanov A V 2006 Models for vector multicomponent physical
quantities and a multivariate test method for optical measurement systems Measurement
Techniques 49(12) 1182-1188
[8] Nesterov V N and Meshchanov A V 2007 Theoretical principles of optical measurements on the
components of multicomponent displacements for mobile objects on the basis of multivariate
tests Measurement Techniques 50(11) 1127-1136
[9] Nesterov V N and Zhmurov D B 2007 Formal synthesis of a data-acquisition system for
multicomponent physical quantities Measurement Techniques 50(9) 903-907
[10] Nesterov V N, Mukhin V M and Meshchanov A V 2008 The way to measure the components of
complex movements of an object Russian Federation patent No RU2315948 URL:
http://www1.fips.ru/fips_servl/fips_servlet?DB=RUPAT&DocNumber=2315948&TypeFile=ht
ml
[11] Nesterov V N, Mukhin V M and Meshchanov A V 2012 The method of controlling the accuracy
of contour movements of industrial robots Russian Federation patent No RU 2466858 URL:
http://www1.fips.ru/fips_servl/fips_servlet?DB=RUPAT&DocNumber=2466858&TypeFile=ht
ml
[12] Nesterov V N, Nesterov D V and Mukhin V M 2017 The way to measure the components of
complex movements of an object Russian Federation patent No RU2610425 URL:
http://www1.fips.ru/fips_servl/fips_servlet?DB=RUPAT&DocNumber=2610425&TypeFile=ht
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 159