The proposed approach for pose estimation is based on the construction of a Convolutional Neural Network with an encodingdecoding structure and a spatial pyramid based on WASP structure in its bottleneck and a Discrete wavelet transform encoder. These techniques already shown their capabilities to solve the main problems in state of the art related to: diferent Field of view (FoV) required to analyze the diferent possible sizes of a specific subject. we want to solve the faulty structure of the modern CNN based Neural Networks in the encoding part using DWT encoder and WASP. This Work also have the objective of demonstrating from a more general point of view which could be the advantages of a Discrete Wavelet Transform (DWT) encoder in any CNN-based approach for Pose Estimation and Object detection in any form, such as for several subjects in the same image or in the internal video due to the almost redundant use of the usual most famous encoding structures for CNN such as ResNet-101, U-Net or VGG16-19. we will do our tests using a U-net Based CNN in order to evaluate the importance of the results of the Discrete Wavelet Transform encoder also in the decoding part through the cropping of theme at the last layers of the network. This is necessary due to the loss of border's pixels during encoding that could be useful for the result's evaluation.
by diferent works in image classification as [ 7, 8] or for multiple subject in the image in which case we not only
object detection approaches as in [1, 9] or other solutions establish a heatmap for the evaluation of the key points
[10, 11]. With this premises our work is focused on the in a Gaussian map but we consider have to also a
segapplication of an U-net based model with a multi-level mentation of the image and evaluation of long,medium
decomposition (MLD) of the image as encoder parallel to and short range ofset for each key point of the subject
U-net with concatenation of the gathered information for to establish the relationship between diferent key points
each layer of the DWT encoder to the encoder of U-net belonging to diferent subject and the ones in the same
and propagation of this information to the decoder. In segmented part of the image. In these cases is also
imthe feature extraction at the U-net’s bottleneck is instead portant to mention as in [18] the WASPv2 version used
used a WASP structure to obtain the resulting feature in combination to HRNet structures without a decoding
map of the image from diferent field of view after the structure that follow it performing a cascade of Atrous
encoding part to pass to the decoder. In this way the convolutions at increasing rates to gain eficiency. In
encoder do not loose the information eliminated by the the next section are explored the past works in DWT
down sampling operation because reconstructed from the application to a variety of state of the arts and explains
information of wavelets passed forward to the decoder. how our approach is diferent in chapter 3 and in the
conclusion how the results are improved by it. Talking
about possible applications of Discrete Wavelet
Trans2. Related Works form (DWT) related to diferent state of the art we can
consider as fundamental the contribute given by D-Unet
Two main aspect of this works are considered as first a dual encoder used for diferent purposes as Image
segrespect to the works related to the wavelet applied to mentation and object detection [19]. DWT-based encoder
diferent fields as Image segmentation or object detection is an important addition to the state of the art because
and then the common structures and modern approaches it demonstrated its superiority in reconstructing
inforactually used in Pose Estimation, this work is focused on mation lost in the encoding part of the neural network
the analysis of approaches from both this two aspects and also the superior capability with respect to diferent
used in combination with pre-processing common prac- methods used as steganalysis rich model (SRM). In which
tise related this to the task of Human Pose Estimation. case the layer extracts the image noise providing
addiThere are many pose estimation systems [12, 13, 14], tional evidence for the classification of multiple types of
among them UniPose [15] is a pose estimation method image key points as shoulders, elbows and faces. In [20]
based on the application of a so-called WASP structure has been instead demonstrated the problems related the
as the central part of the bottleneck in the CNN. It is use of a down-sampling and up-sampling operation and
based on the use of layers with diferent dilation in the corresponding interpolation operation needed to
reconapplication of a rate parameter related to the formula: struct the original image from the global feature map at
1 2 the start of the decoding part. As last consideration is
[, ] = ∑︁ ∑︁ [ − , − ] * [, ] (
As mentioned in [21] the Fully Convolutional Net- in front on the camera are superior but when the limbs of
works (FCN) are the most used kind of CNN in this the subject are composed by very small groups of pixel
ifelds and all are structured as encoder-decoder with a more pixel-by-pixel analysis is needed. In terms of
up-sampling procedure for reconstruction of a resolu- result we obtained a level of PCP for 50 epoch-training
tion and restore of loosed data in encoding part. In this showing the challenging properties of the images. The
section are considered as assumption that the structure parameters modified are in fact not only the kernel sizes
will be similar to U-Net. however is important to remark but also the dilation or rate parameters obtaining in this
that these kind of structures already establish state of way a general FoV of the image. We also tried to use
the art result without thorough consideration of other diferent sizes for the Latent representation, needed to
methods of image feature extraction. As first approach use the dilation high and use bigger FoV than we can and
has been considered a solution based on the generation more parallel levels of FoV concatenated for the decoding
of an heat map but in that case the construction of it is part. The shapes of the layers variate between 1,2,6 rate
very similar to the binary segmentation of [19] which parameters while the dimension of the kernels between
can lead to problems as the necessity to use heat maps 3,5,7. Another important fact is related to the presence of
separated for each key point in order to connect theme the 1 by 1 convolutions before the concatenation useful if
each other in a correct way even if we have overlap of we want to manipulate the dimension of the data without
it. So in this case a simple binary segmentation is not loss of features from the local to the global feature map.
useful for overlap of limbs and articulations and a more
computational complexity shows up in working
separately for each key point. For these reasons has been 4. The DWT-based Encoder
chosen to consider a layer for a regression task with a The methodologies applied for the construction of the
invector representing the scaled coordinates of the points formation related the analysis of the image in frequency
in the image that we have to interpolate with our model. with diferent scale lead me to many diferent choice,
The reason behind the U-net structure as choice, instead from the application of Gaussian or Sobel Filters to the
most common VGG or ResNet, is due to the possibility to use of SRM structure as in [22]. But what in the end
propagate the information produced with wavelet’s coef- establish the most significant result has been the DWT
ifcients everywhere in the layers between encoding and encoder with the multi-scale decomposition of the image.
decoding. The network to propagate context information To make it we built a sequence of layer applying low
to higher resolution layers, exploiting this capability of pass and high pass filters to an image and generating
Unet the information propagated are in the structure of relevant Haar-features for the localization of relevant
our Neural Network the information gathered from the key points.The Multilevel decomposition method called
wavelet in the encoder layers concatenated layer by layer in this work DWT-based encoder will generate these
coto the information obtained from the network’s encoder. eficients over all three direction in the image vertical,
This will lead us to a distance function weighted respect
to the subject dimension in the image that we will have cdoiaegficoiennatls afonrdthheodrieztoaniltsaaln(dapp,roxim,ationo,verd )iferw-ith
to minimize (e.g simple MSE respect the position) and ent thresholds to pass to the next layer as explained in
ifnd the best function that interpolate the position of the [23],[24] or [25]. With a more mathematical viewpoint
key-points respect the image’s information. We want in each frequency component can be defined in a matrix
this way to be able to build augmented information for form for 2D input as:
an image with very low dimension and use them to infer
invisible information for a simple U-net encoder-decoder. (
: Γ = ⎜⎜ − 1
⎜ − 2 − 1 0 ⎛ ... ⎜⎝ ...
...
... 0 1 ...
... 1 2 ... ... ...
3 1 2 ...
...⎞ ...⎟ ...⎟⎠ ...⎟⎟, K = ⎜⎜
⎛ ...
⎜ ⎜⎝ ...
... − 2 − 1 ... 0 1 ... − 1 0 ... 1 2 ... ... ...
3
These information produced for each layer will be con- It can be interpreted as a Fourier transform of f at the
fre(
(
−∞
quency , localized by the window g in the neighborhood
of . Multiplying the signal represented by f(t) with g and
computing the Fourier coeficients we obtain indication
of the frequency content of the signal f in a
neighborhood of , shifting the window from 0 and obtained a
(
("coherent states"), an important property of this
func(
(
= translation where the resolution of the identity is written as: = − 1 ∫︁ ∫︁ 2 ⟨ (, ), ⟩ (, )
(
]
(
IDWT in decoding for obvious reason and the fact that, having a simple Multi level decomposition without variation filters. We will have just one gradient in common wavelets applied as Daubechies that already proved their image using: to consider for the loss minimization evaluating the ad- as direction rewriting (, ) as ,. Given a function ditional DWT features directly in the same CNN’s loss as in the case of [15] with diferent loss for each heat () as signal of input that will be our image, it has a large amplitude near sharp transitions of pixels such as edges, obtaining the coeficients over the ⟨ ,, ⟩ ≥ threshold and varying it over the frequency. What now is produced by this are three corresponding high pass and low pass filters we obtain four results As approximation and details coeficients for each of the three layers to pass to the next one in the multi level decomposition where each of theme is defined as: ⎧2 = ( 2 (− ) 2 (− ))(2− , 2− ) ⎪ ⎪ ⎪⎨21 = ( 2 (− )˛2 (− ))(2− , 2− )
(
but in the end, the evaluation has been done on a
combination of it.The results in accuracy are evaluated not from
the first epoch but from the 50th epoch while the loss is
2 = ( − )2 + ( − )2 (
the PCP that is based on detected joint that is considered correct if the distance between the predicted and the true joint is within a certain threshold. In this work you can find an example of the results in terms of accuracy nonised images. The images have been chosen to give a basic representation of the main problems of the human body pose estimation due to the complexity of the pose and occlusion of the limbs of the subject. Looking at the Figure 2: Test with HaarDWT used with pretarined CNN ifrst line the first picture, beyond a small negligible error, Daubachies. denote a very good result in the pose estimation of the complicated CNNs are not capable to compete with the subject. Diferently, the third image is more complex to wavelet encoder showing clear results of over fitting so evaluate due to the occlusion of the limbs and the com- we preferred to do not modify epochs more than 100 and plexity of the pose. A diferent kind of problem is instead using just 4 convolution layers. We extended these result represented by the pose of the second and third image even to a diferent topic as human pose estimation, the where the entire image down sampled with a very high initial objective was to establish results in object detecfactor leads to the problem of low border resolution of the tion approach for pose estimation but it has been discard subject giving imprecision in the evaluation of the key because as already said the top-down approach proved points positions. Note carefully the presence of padding to be superior. In addition to this we solved the faulty in the CNN leads us to the shifted results in the figure as encoder-decoder general structure common to all most in object detection as [28]. In an other important test we used CNNs for this field as VGG-16 and ResNet showconsidered the DWT based method confronted between ing lower loss of information during encoding using the COCO and LSPII dataset where the data are more prob- DWT and the analysis of the Wavelet’s information of lematic and many subject assume complex poses. These the images. tests are also evaluated with respect a PKC value but in the end we used the PCP because more appropriate for a bottom-up approach and easier to implement and References evaluate but similar results are initially evaluated with respect the PKC.
The wavelet procedure actually increase the capabilities with the DWT encoder and with this result how is possible to extend it also to diferent fields that include these kind of structures. Diferent results are obtained using more epochs or layer in the convolution structure from 4 to 3 layers obtained lower results in PCP terms so more
ing a machine learning approach with gan-based data augmentation technique trained using a custom dataset, OBM Neurobiology 6 (2022). doi:10.
Method + ,ℎ ,ℎ CNN+ ,ℎ U-net , ,ℎ