Geometric image distortions appear when cameras register the image from behind a refractive object - e.g. a car windshield. To ensure the reliability of 3D perception algorithms, a distortion model is necessary. The model has to be general enough to capture the variety of possible refractive object geometries. We propose a method where we directly model the refractive media as a thick ellipsoid, and compute the resultant distortions by tracing individual light rays as they refract on the inner and outer surface of the object. With this new ellipsoid model provides flexibility and via the model parameters we are able to capture all important factors influencing distortions, namely the curvature of the surfaces, position relative to the camera, and thickness of the refractive material. We test the proposed model on a synthetic dataset, analyzing the advantages and possible failure cases of our method.
Video-cameras are preferred sensors for perception in robotics and autonomous
driving because of their low cost and high resolution. When vision is used for 3D
perception [
Geometric distortions can arise when the camera is placed behind a protective
cover – e.g. a windshield of the car. These distortions are afected by the global
properties of the object – e.g. position to the camera, curvature of the surface
and thickness of the material – as well as irregularities of the surface, resulting
in local distortions. In our previous work [
In this work we use a similar methodology as in [
Calibration methods can be classified as static calibration methods [
Diferent algorithms use specific distortion models with diferent complexity.
The most widely used models consider radial distortions [
In our work we consider distortions from light refractions. Similar work was
done by Agrawal et. al. [
Deep learning methods are also used to estimate radial distortions [
We model the refractive object as the space between two ellipsoids. The two ellipsoids have the same center position ∈ R3 and orientation – represented as an axis-angle rotation ∈ R3. The inner ellipsoid has the semi-axes , , ∈ R. To define the semi-axes of the outer ellipsoid, we add an additional, small thickness ∈ R to each semi-axes of the inner ellipsoid. This way + , + , + ∈ R give the semi-axes of the outer ellipsoid. The quantities = { , , , , , } together form the parameters of the ellipsoid object model.
To make further computation simpler, it is useful to see the ellipsoid as an afine image of a unit sphere centered at the origin. The transformation is defined by a 3 × 3 matrix and the translation vector , with = ( )diag(, , ), where ( ) is a rotation matrix constructed from using Rodrigues’ formula, and diag(, , ) denotes a diagonal matrix with values , , . Using the afine mapping, all necessary operations, including intersection with a ray and surface normal evaluations can be reduced to operations on the unit circle.
The scope of a camera model is to associate pixels with light rays from the outside world. The intersection points between these outgoing light rays and objects (e.g. a checkerboard pattern during calibration) will define what we see on the image. In a distortion-free setup this outgoing light ray coming from the camera center and going through the image pixel – this is the ray given by the pinhole camera model. In our physical model the direction of the original light ray is modulated when it enters or leaves the refractive object – in this work modeled by an ellipsoid. The change of direction is computed using Snell’s law of refraction, and it is a function of the incident ray, the surface normal at the intersection point, and the relative refractive index of the two materials (e.g glass and air). The complete raycasting process is fully diferentiable, allowing the gradient based optimization of the ellipsoid model parameters. The method is implemented in the PyTorch framework to leverage the backward mode automatic diferentiation for the optimization.
The ellipsoid model of the refractive object overparameterizes the image distortions, resulting in arising symmetries of the physical model. The observed distortions are invariant with respect to a set of transformations applied to the object. As a result of the invariance, when we invert the model to estimate parameters of the ellipsoid, the full parameter set is not recoverable without some prior knowledge about the object. While in some cases an estimation of the distortions may be suficient, reconstructing an approximate 3D model of the object can also be desirable. Identifying the symmetries and properly handling them using regularization techniques is therefore an important step in our method.
To give an intuitive example of these symmetries, we consider a simple 2D case where the refractive object is a thick circle, and we consider two variables: 1) the 0.200 ) 0.175 ree 0.150 (g e d 0.125 r
or (b) 000...001570050 anangleer 0.025 Me 0.000 0.0 2.5Cen5t.e0r X7c.o5or1d0in.0at1e2(.c5m1)5.0 17.5 20.0 (a)60 55 50 R4a5diu4s0(c3m5) 30 relative distance of the circle center to the camera center in the direction of the optical axis; 2) the radius of the circle. This analysis translates well to the center position of the ellipsoid and the length of the semi-axes, the two sets of variables where these problems arise in the 3D case.
We consider the reference parameters of circle position of −10 cm and circle radius of 40 cm, and compare all other distortions to the ones measured in this configuration. We express the distortion error relative to the reference setup as the angle between a refracted ray in the reference setup and the examined setup, averaged over multiple incident rays with diferent initial orientations.
Figure 1 shows the distortion error relative to the reference setup for diferent parameters. The black notes a set of parameters where the distortion error is less than 10−3 degrees. We can observe that we can achieve low distortion errors as long as we adjust both parameters simultaneously in the proper way.
We use the standard setup of static camera calibration to estimate the model parameters. Given a pixel, our forward model finds the 3D point in the scene seen by the camera. In a static calibration setup we use a planar checkerboard pattern as the target object, where the dimensions of the squares are known. Using model inversion – based on a set of images of checkerboard patterns – the parameters of the ellipsoid model can be recovered using gradient based minimization.
Our loss function has two components: a reconstruction and a regularization term. Let be a calibration image, where , represents the pixel locations of the th checkerboard corners on the image. We denote with , (·) the raycasting function which takes a pixel and returns a point on the checkerboard object associated with image . Let ,
be the ground-truth world coordinates of the checkerboard corner associated with pixel , . Then the reconstruction error is expressed as the squared error between the estimated and ground-truth corner coordinates: ℒ ( ) = ︁∑
︁∑ , ⃦ ⃦ , ︀( , ︀) − , ⃦⃦2 .
In Section 4 we showed that the ellipsoid model overparameterizes the distortions, and the full physical model cannot be recovered without prior knowledge. We include this prior knowledge as a regularization term during the minimization. More specifically, we constrain the distance between the camera center and the point where the principal axis of the camera – also being the Z axis of the camera coordinate system – intersects the inner surface of the ellipsoid. Let ( ) be the distance between the camera center and the ellipsoid, and be the expected constant value of this distance. Then the regularization term is a 2 penalty: The full loss function is a weighted sum of the two terms:
ℒ ( ) = ( ( ) − )2 . ℒ ( ) = ℒ ( ) + ℒ ( ). (5.1)
The loss function in Equation 5.1 is minimized using the L-BFGS optimization method.
We chose the L-BFGS method as it is a quasi-Newton method, which is both eficient and it does not require a direct evaluation of the Hessian matrix, making the method compatible with backward mode automatic diferentiation.
We tested our method on a synthetic dataset. We generated a set of ground-truth ellipsoid objects, and we used our forward model to render images with checkerboard patterns. The simulated camera mimics the properties of a Raspberry Pi camera module, recording images at 3280 × 2464 pixel resolution with a horizontal ifeld of view of 62.2 degrees. For each run we used 10 checkerboard patterns placed therefore the 10 images provided 640 data points in total. at random positions and orientations. Each pattern had 8 × 8 corners on them,
After generating an image dataset, we reinitialized the parameters of the ellipsoid to represent a small circle centered and the origin. We used our optimization method presented in Section 5 to find the optimal parameters, ideally corresponding to the ground-truth ellipsoid used to generate the samples. The minimization method was able to find a model with sub-pixel distortion error in each case. The resulting shape of the ellipsoid however did not always match the ground-truth model. a), b) and c) the estimated ellipsoid approximates the surface well on the area which 40 Z20(cm) 0 20 2015105 0 5101520
X(cm) 8.7 3. 0 40 30 20Z (cm1)0 0 10 20 302010 0X1(c0m20)30 is seen by the camera. On the other hand, we can also see that in some case (e.g. in example b) this good local approximation can be achieved without finding the global ground-truth parameters. This suggests, that the exact estimation requires more prior information about the refractive object. Although the predicted ellipsoid is not always correct, the distortion errors are under one pixel in each example, with small variations across the image. Example d) shows a case where because of not using regularization, even if a solution with small distortion error was found, the predicted ellipsoid is not approximating the real surface well, not even at the area seen by the camera.
We presented a distortion estimation method for scenarios where the camera is placed behind a refractive object. We modeled the object as an ellipsoid, and used machine learning techniques to estimate the model parameters. We analyzed the possible failure cases, where multiple diferent ellipsoids result in the same distortions, and proposed a regularization which solves this issue. The method was tested on a synthetic dataset, generated using the forward model of image generation, implemented as a raycasting algorithm. We were able to obtain a close approximation of the object surface on the region seen by the camera.
In our future work we will focus on the validation of the algorithm in a real
scenario. Although our method achieves good results in a noise-free setup,
application with real cameras and dataset is still an open question. We will also look
into the integration of the local model in [