Development of the autonomous underwater vehicle technical vision systems is impossible without precise debugging and testing. Due to the factors described in the paper, such testing in most cases cannot be implemented with the help of real AUV. Therefore, the usage of the procedurally generated virtual testing areas is suggested. The algorithm for generation and visualization of the seabed 3D model that is suitable for AUV technical vision systems testing and debugging is described. This algorithm allows building of high detailed surface of the seabed, where each part is absolutely unique and does not contain repeating texture patterns. Also, software system “AUV Vision Debugger” that consists of seabed generator and AUV simulator is considered. The simulator provides interaction between generated seabed, AUV model, and technical vision system that is currently being tested.
Complexity of the underwater navigation and inability to organize communication with high capacity data exchange between autonomous underwater vehicle (AUV) and operator leads to necessity of onboard technical vision systems development to improve underwater vehicle autonomy. High reliability and quality operation in various conditions is required from such systems. Their development and further usage is impossible without precise debugging and testing. Currently testing process organization and testing data obtaining becomes a serious problem.
At present time, testing areas with markers and targets located on seabed and specifically equipped pools are used for tasks of testing and debugging. Such methods are well suited for testing AUV devices and equipment, but they are not applicable for AUV technical vision systems testing and debugging because of following reasons. 1. Obtaining data from AUV is a time consuming and expensive process. It includes not only mission accomplishment itself, but also transportation to testing area and device launching that is unacceptable especially at the early stages of development. 2. It is not always possible to obtain necessary data for testing from areas with different types of relief due to geographic reasons. 3. It is impossible to evaluate vision system accuracy while using data from real testing areas, because there are no other ways to measure seabed with the required accuracy level (up to cm) on large areas. 4. It is impossible to interfere in the vision system work at any required time. 5. There is a risk to lose the AUV due to technical vision system errors. To avoid all mentioned problems of real data usage in AUV technical vision system testing and debugging processes, it is suggested to replace real experiments with tests in virtual environment. The software system “AUV Vision Debugger” that allows holding such tests is considered in the paper. 1
Usage of the virtual simulations for tasks related to AUV is an actively
developing research area. Currently a lot of modeling software for different purposes
is developed. The workbench [
Generation and visualization of the seabed 3D model, that has good enough
quality for AUV vision system testing and debugging, is the most complex task.
Despite large number of existing methods for landscape generation [
Structural scheme of the developed program system for AUV technical vision algorithms testing and debugging – “AUV Vision Debugger” is shown in Fig. 1. This system consists of two programs: seabed 3D model generator and AUV simulator. The generator is a GUI program that allows building a seabed 3D model from the set of user defined parameters using approaches described in section 3. The simulator is a program that simulates AUV dynamic in virtual environment and allows user to observe all AUV movements by controlling the view camera. The physical model used by simulator is described in section 4.
One of the simulator’s capabilities is a selection of the controlling program – technical vision system combined with control system. The procedure of vision system testing and debugging may be different for different development stages and tasks that system must solve. Therefore, specific control system is required to implement testing of concrete vision system task. “AUV Vision debugger” does not impose any limitations on organization of technical vision and control systems and their interaction, and provides universal interface with simulator.
The simulator runs the controlling program and scans its standard output stream with approximately 50 Hz frequency to detect passed commands. As an answer to the commands the simulator writes messages to the standard input stream of the vision system and saves graphical information in the file system if required.
Commands supported by simulator can be divided into three groups: control commands, state requests and image requests. Control commands serve to change such states of AUV parts (see section 4.1 for parts description) as position, orientation, power and etc. State requests can be used to know information about AUV position, orientation and velocity, distances to objects (if AUV equipped with sonar) and etc. As an answer to state requests, the simulator passes state information to the standard input stream of AUV technical vision system. Image requests are necessary for obtaining images with debug information from AUV onboard cameras. Images and debug information are not passed directly through standard input stream, but stored in file system. Instead, a notification that images are ready to use is passed to the input stream. The debug information contains accurate camera external and internal parameters and distances from camera to image pixels. Using this information, the controlling program can evaluate results produced by the technical vision system and make conclusions about the system’s accuracy.
Thus, “AUV Vision Debugger” provides just a physical and visual model of environment and AUV. Development of the control system for testing AUV technical vision algorithms is the user’s responsibility. 3
The algorithm that was developed for “AUV Vision Debugger” to generate and visualize the seabed 3D model has five stages: generation of low frequency relief map (section 3.1), fractal noise computation to increase level of details (section 3.2), 3D model mesh building (section 3.3), model refinement during visualization (section 3.4) and texturing (section 3.5). 3.1
Relief map H = ( ), = 1, 2, ..., ℎ, = 1, 2, ..., , where and ℎ – map’s size, defines relief height in point ( · , · ), where is map resolution specified in meters per point (m/p). The maps with = 10 m/p and , ℎ 6 1000 were used in the “AUV Vision Debugger”. Such maps can describe the seabed with length up to 10 km that is more than enough for AUV technical vision algorithms testing and debugging.
The relief map contains such basic seabed elements as coast, shelf, continental slope, ocean bed, submarine canyons and mountains (Fig. 2a). These elements are defined by user through setting a few parameters’ values, and they do not require specific artist knowledge.
Contours of the basic elements are specified with fractal lines and heights are interpolated, using different interpolation functions, based on distances to defined contour lines (Fig. 2b). 3.2
The relief details that have frequency more than 1/(2 ) m−1 cannot be represented by H map. Meanwhile, to provide correct work of technical vision algorithms on distances from seabed that could be reached with onboard light equipment, accuracy up to millimeters is required. To improve relief level of details we will use fractal noise that consists of summed Perlin noise functions, taken at following frequencies: 2 −1
, f = ( ) = fℎ = ( ℎ ) = 2 + where = 0, 1, ..., , = 0, 1, ..., ℎ , = ⌊log2(2·
– drawing fragment size (pixel of the screen or AUV camera) in meters. If fragment size could not be computed, the desired accuracy should be directly assigned to . Low frequencies f will be added to seabed 3D model and high )⌋, ℎ = ⌊log2( −12− ) , ⌋ frequencies fℎ will be utilized during visualization process.
Amplitudes on the same frequencies may be different for different seabed parts, depending on their slope roughness and some random factor. Let S = ( ) be the slope map (Fig. 3a) and R = be the roughness map (Fig. 3b), where = 1, 2, ..., ℎ , = 1, 2, ..., , = min 1, ︂( ( ) ︂) = 2 arccos( , ) , , where ( ) is standard deviation of high frequency part of height map in 9 × 9 neighborhood (weighted with Gauss function with = 2) of the point , is a normalization factor and n = ( , , , , , ) is a normal to low frequency part of height map in point .
We divide the frequencies range (f , fℎ ) into three groups each of which is controlled by own basis function ( ), = 1, 2, 3, – Fig. 4a. The logarithmic scale is used on the charts; at this scale functions ( ) are piecewise-linear. As well we introduce functions of roughness and slope influences on the 1st and 2nd frequency groups:
I 1( ) = 0.45 2 + 0.05, I 1( ) = ︃{ 6 2
− 8 3,
(8 − 2)( − 1)2, > 0.5
ence function for -th frequency group that unites roughness, slope and random
factors:
(x) = max (0, min (1, I (R( 1, 2)) + I (S( 1, 2)) + P (x))) ,
(1)
where x = ( 1, 2, 3) is a space point, = 1, 2, R( 1, 2) and S( 1, 2) are the
user defined coefficient of the random influence on -th group,
values obtained from R and S by means of bilinear interpolation, ∈ [0..1] is
P (x) =
︁∑
4
(M x + T ) ,
︂)
where = 1, 2, 3, P(x) is a Perlin noise function [
A(x, )P( x).
(3) ︁∑ =0 3.3
For mesh construction we will use approach [
It is necessary to tessellate mesh during visualization to provide size close to one pixel for all visible triangles. The most of modern GPUs supported hardware accelerated tessellation with subdivision by up to 64 parts for each triangle side. Such subdivision will not be enough for mesh triangles located near the camera. Therefore, tessellation based on precalculated triangles sets (PTS) is suggested for these triangles.
Each triangle side could be divided into 2 parts, where = 1, 2, 3 is an index number of triangle side (counterclockwise numeration), and = 0, 1, ..., . AUV operation distance to seabed and onboard cameras resolution allows to choose = 9 that leads to = ( + 1)3 = 1000 of different variants of tessellation. Let us consider -th tessellation set, = 1, 2, ..., , that consists of triangles. Each triangle of -th set is described by three vertices v = ( 1, 2), = 1, 2, ..., , = 1, 2, 3 (counterclockwise numeration). Values 1 and 2 are coordinates of -th vertex of -th triangle of -th set, they defined in the coordinate system with basis vectors represented by 1st and 3rd sides of -th triangle and origin in the intersection point of these sides – Fig. 6a.
Using such coordinate system we can draw -th tessellation set instead of some mesh triangle that have coordinates p1, p2, p3, by projecting local -th set coordinates to the global space: x = 1(p2 − p1) + 2(p3 − p1) + p1. (5) The fact that tessellation sets store coordinates independent from mesh coordinates allows us to put these sets into video memory once, and then just use through one function call. The attempt to draw a triangle is replaced by suitable tessellation set drawing under control of vertex shader that computes global vertices coordinates using (5). Coordinates p1, p2, p3 are passed to vertex shader as uniform parameters. The number of tessellation set is defined based on triangle size and its remoteness from the camera to provide pixel accuracy in the screen space.
To speed up visualization process the seabed model is divided into rectangular blocks, so that blocks number and triangles number per block would allow iterating through them before each frame rendering. Blocks located behind camera’s clipping planes are discarded. For rendering of the triangles located in blocks that are close to the camera, the PTS-based tessellation is used. If the block is far enough for using hardware accelerated tessellation only, it renders by single drawing function call. The process of seabed subdivision is shown in Fig. 6b.
After tessellation the fractal noise (3) is added to each vertex in the direction of the interpolated normal n, computed in this vertex:
x^ = x + n (N(x, ℎ ) + (x)) ,
where (x) is a density function (4). The example of tessellated and amplified
with high frequency noise seabed model is presented in Fig. 7.
Texturing of landscapes, developing for testing AUV technical vision system has
its own specific:
1. Only procedural textures could be used, because using of the bitmap images
leads to the appearing of repeated texture patterns that makes impossible
the testing of the algorithms based on feature points [
Surface type and, respectively, texture type is defined based on functions 1(x) and 2(x) (1). The sand procedural texture with weight is mixed with the stone procedural texture with weight 1 − , where (x) = ∏︁ =1,2 (1 − smoothstep( − , + , (x))) .
Smoothstep is a standard GLSL function. Constants define boundaries between relief types and is a transition width. These constants are set during seabed generation. The Fig. 9 is obtained with 1 = 0.2, 2 = 0.12 and = max(0.0025, 0.05 ), where – is a drawing fragment size in meters.
Creation of procedural texture for each type of surface requires individual
approach. Sum of Perlin noise functions or cellular basis [
The main tasks of the simulator are visualization of seabed and AUV models, simulation of AUV dynamics, and computation of interactions with the environment. Simulator also provides the possibility to observe the mission process. “AUV Vision Debugger” was created mainly for testing and debugging of technical vision system and not for testing AUV control systems and devices, therefore simplified AUV model (section 4.1), its dynamics (section 4.2) and interactions with seabed (section 4.3) are used. AUV model consists of parts, described in text file of the model. Each part has its own type: shell, engine, control surface, sonar, floodlight and camera. In the model description there could be only one shell and arbitrary number of parts having other type. Let us consider main parameters of different types: 1. Shell is defined by its weight and 3D model in 3DS format. The coordinate system of the shell is a basis for all other parts. 2. Engine and control surface are defined by their weights, 3D model, transformation matrix, describing position and orientation relative to shell. Directions and ranges of engine and control surface allowable movements and rotations are also defined. Thrust vector and its magnitude range could be defined for engine. 3. Sonar, floodlight and camera do not have their own weight and graphic representation. They defined by position and orientation relative to shell. Directions and ranges of allowable movements and rotations are also defined. Light power could be set for floodlight and focus distance, resolution, radial distortion, and lighting dependent errors could be set for camera. Apart from mentioned parameters, each part has a name that allows the vision system to interact with this part. 4.2
From position of dynamics AUV with all its parts is considered as one rigid
body. To calculate its motion Newton–Euler equations are used. The details of
motion computation process based on forces and torques acting on the rigid
body are well described in [
Simulator takes into account following forces: gravity force F , thrust force of -th engine F , = 1, 2, ..., , where is engines number, pressure force F and hydrodynamic force Fℎ . All forces except F induce torques relative AUV center of mass, that accounted in motion calculation. Let us consider how these forces act.
F = (0, 0, −
Gravity force is applied to AUV center of mass and pointed vertically down ), where
is sum of weight of all parts, and is acceleration of gravity. Thrust force F
is applied to -th engine center of mass and directed along thrust vector, described in AUV text file. If rotation is applied to -th engine then its thrust vector is also undergo this rotation. Magnitude of F is chosen by vision system from the range defined in model text file.
Forces F and F
ℎ are calculated for each face of AUV 3D model if this face has nonzero area , external unit normal vector n and located out of the shell. Forces F and Fℎ are applied to geometric center of the face c = ( 1, 2, 3): where is a water density, v is velocity of c relative to environment, F ℎ and F ℎ are components of Fℎ in direction of v and normal to the v vector direction:
F = − | 3| n, F ℎ = ℎ
+ ℎ −v v | | n × v × v |n × v × v | , ℎ = |n · v| |v| ,
2 ℎ = √
n · v |v| 2 ︃√ 1 − ︂( n · v | | v )︂ 2 .
Presented forces are sufficient to describe AUV dynamics accurate enough for technical vision systems testing and debugging. 4.3
Collision of AUV parts with seabed surface is a very dangerous situation that should be necessarily prevented by a technical vision system during visual navigation. Therefore, it is very important to detect such situation and to inform the vision system in the case it occurs. Informing is implemented by passing a message to the standard input stream of the vision system. We consider a process of detecting collision by the simulator and its reaction on this collision.
Collisions are tested on each simulation iteration for parts points. Let us consider collision detection process for point x. The collision is tested for all seabed model triangles that are closer to x than one meter. If there are no such triangles then point x is above the seabed. Let p be the vertices of triangle that is closer to x than one meter, = 1, 2, 3, and n be the normals to the model in these vertices. To detect a collision we should perform following steps: Step 1: Computing face normal: n = (p2 − p1) × (p3 − p1).
Step 2: Computing intersection points between rays from p in directions n and plane (n, −n · x): p′ = p + n · x − n · p n · n n . Step 3: If x does not lie inside triangle (p′1, p′2, p′3), then no collision detected, else go to step 4.
Step 4: Finding of interpolated unit normal n˜ in x point of triangle (p′1, p′2, p′3) with vertex normals n , by means of bilinear interpolation.
Step 5: Computing point of triangle (p1, p2, p3), corresponding to x:
If > 0, then x point is located inside seabed, therefore collision happened. If value > 0 computed for more than one triangle, maximum value should be chosen.
If collision of depth for some point x with normal n˜ is detected, then following force is applied to x:
F = ( − v · n˜)n˜, where is an elasticity coefficient, is a damping coefficient, and v is a velocity of x point. Application of F force prevents AUV penetration into the seabed.
Methods and algorithms, suggested in the paper with developed program system “AUV Vision Debugger”, allow testing and debugging of AUV technical vision systems in virtual environment that leads to following advantages: 1. high speed and low cost of testing data acquisition, 2. ability to obtain testing data from different surface types, starting from sand valleys and ending with rocky canyons, 3. ability to evaluate technical vision accuracy since the investigated seabed surface is precisely known, 4. ability to interrupt system working exactly at the moment when error occurs, 5. testing result repeatability.
Currently AUV Vision Debugger allows getting high detailed images of seabed, all parts of which are absolutely unique – Fig. 9. For further researches it is planned to add a possibility of procedural modeling and visualization of marine flora and fauna, including dynamically changing objects, to add more procedural textures and submarine caves network. It is also planned to learn ways of integrating our work with existing general purpose software modeling systems.
It is evident that using of synthetic tests in “AUV Vision Debugger” does not allow us to completely abandon the real experiments, but it significantly reduces their amount. As a result, the time required for AUV technical vision system development is decreased and reliability is increased.
Acknowledgments. This work was supported by Russian Foundation for Basic Research (grant 16-31-00187 mol a).