Digital Holography is an emerging eld of a new paradigm for general, as well as microscopic imaging applications. The goal of the project is to conduct experiments for the processing of digital holograms using deep neural networks as an approach to AI-based/semantic large scale and complex information and content analysis. With the usage of deep neural networks, this project develops new techniques for a more robust and performant processing of digital holograms, especially for the task of numerical hologram reconstruction. We aim to explore the semantic segmentation network models, and challenge to improve the performance of processing holograms. Among the most popular network models, the architecture called U-Net was chosen to achieve this task. In these works, several experiments were conducted applying this method to arti cially generated holograms, with promising results. These positive results lead to the believe that a signi cant improvement could be made on real holographic data, that could lead to achieving a more robust hologram reconstruction.
1.1
recreate the object, the hologram is recorded digitally and reconstructed. To perform this, the amplitude and phase of the object waves are computed based on the di raction theory of light. This derivation of the wave front in a particular plane is called numerical reconstruction. 1.2
Project Motivation For the automatic processing of holograms in DH, two main tasks are of interest: nding the right reconstruction distance and performing the numerical reconstruction for a given hologram. Usually, both tasks are performed separately and may result in very long processing times. In the rst phase, it has to be evaluated if these two tasks can be performed by a dedicated deep neural net. If each problem can be solved by a neural net, the idea is to solve both problems with a single pass through a single neural network.
The main goal of the project is: { Detection of particle in streaming water. { To explore if deep neural nets can be used to process digital holograms. { To evaluate if the usage of U-Net will speed-up the process of holography and increase overall performance and accuracy.
We proposed a modi ed U-net model which can improve the performance of a wide range of images. We showed promising results on both computer-generated and real hologram datasets. Figure 1 shows the vision of the project. 2 2.1
Loss Functions In any learning network model, an error of t is estimated to decide whether the model is a good t or not. The error is, the di erence between ground truth g and the predicted output p.
L(x) = 1 Xn(gi
pi) n i=1 The function which is used to evaluate error is called loss function L(x). There are different loss functions applied for di erent types of tasks. Among which the best tted loss functions for the tasks will be identi ed. It is a backbone of the algorithm to train the network. The goodness of the model depends on the range of error in the network, which is calculated by loss functions.
In this task, the loss is evaluated from the image pixels, which are compared between ground truth object and predicted the output. Currently, for the proposed network model, which is working on di erent datasets: arti cially generated holograms and real holographic data, there are few interesting loss functions. Those are Cross Entropy, Huber Loss, Mean Squared Error and Mean Pairwise Squared Error.
Cross Entropy: It measures the performance of a model, whose outputs are a probability value between 0 and 1. This loss function is also called log loss or sigmoid cross entropy because it applies with the sigmoid activation function. The formulation of loss is de ned as:
C = 1 X [glog(p) + (1 n x g)log(1 p)] where n = total number of items, x = sum of training inputs, g = ground truth, p = prediction.
Huber Loss: It is used in robust regression problems. It inherits the properties from
two popular di erent loss functions: Mean Absolute Error (MAE) or L1 loss and Mean
Square Error (MSE) or L2 loss [
L(a) = (
12 (g (g p) p)2
Mean Pairwise Squared Error(MPSE): MPSE is similar to Mean Square Error
(MSE) [
L(a) = 1 Xn jp(xi) n i=1 g(xi)j2 Where n is the number of pixels in the ground truth, p(x) is the prediction output, g(x) is the ground truth. 2.2
U-Net Architecture
U-Net stands for Unity Networking. U-Net is one of the most popular end-to-end
Autoencoder networks for semantic segmentation. It was developed and rst used by R. Fischer
for biomedical image segmentation at the Computer Science Department of the University
of Freiburg, Germany [
The main goal of this network is to learn to compress the data from the input layer, into a short code manner and then to compress it into a size that is similar to the original image used as input. Encoder gradually decreases the spatial dimensions of the input image, and the decoder gradually recovers the object information and its spatial dimensions of the image. For this decoding, the recovered objects information, have shortcut connections. It is purely based on traditional convolutional neural networks. In this network, the main advantage is when upsampling will be, to learn deeper and concatenating the feature resolutions from the down part. 3
In the book, Digital Holography Pascal Picart and Jun-chang Li state the fundamentals
of holography [
In the article Practical algorithms for simulation and reconstruction of digital in-line
holograms Tatiana Latzchevskaia and Hans-Werner Fink [
In his article, Yair Riverson et.al.[
Olaf Ronneberger and his team at the University of Freiburg implemented a new
approach especially applicable for reconstructing the medical images of objects which
are smaller than 50 micro millimeters [
Description of Dataset Data preparation is an important step in any learning network model. The quality of input data strongly in uences the results which are produced by the model.
As part of the baseline System, we used holograms that are arti cially generated by a computer. These holograms are generated for three simple object shapes rectangle, triangle, and star. Each hologram was generated with varying rotations and object sizes and for a broad range of propagation distances. The gure 2 shows the generated hologram samples.
(a) Hologram of a Triangle (b) Hologram of a Star (c) Hologram of a Rectangle
Fig. 2: Generated Holograms of Triangle, Star, Rectangle.
After successfully training the baseline systems, We used two di erent datasets di raction and real di raction datasets for training real holograms. Both datasets contain generated holograms, however, they are based on real recorded holograms. The two reasons for using generated holograms are: it was not possible to record and especially annotate a huge amount of holograms, and data was needed which was much closer to real recorded holograms for conducting experiments. 4.2
More Realistic Data The di raction dataset was used to train the model in the rst experiments. As mentioned above, this dataset consists of generated holograms based on recordings of real holograms. The process starts with the input of a given real hologram, cropped to a xed region of 256x256 pixels. Afterward, a reconstruction step is performed with the help of a standard reconstruction algorithm so-called Fresnel algorithm and a binary mask is applied to the reconstructed image. The binary mask marks are region containing the object of interest. Using the binary mask, the object is cropped which is done to remove all the background noise hence then left out only with object shapes. Afterward, various geometrical transformations are performed to the object e.g. rotations, translations. The transformed input is fed into the hologram generator which will generate holograms for lots of di erent propagation distances. This process results in a large dataset of hologram images and their respective ground truths. The total process is depicted in the gure 3.
From the rst set of experiments gave us good results on the training set, the test set as well as the blind test sets. The results of real holograms were still not satisfying. That is why we implemented di erent strategies to solve this: we applied di erent levels of noise to the dataset to see if this helps to generalize on real holograms and to improve the hologram generation of even more realistic holograms. The holograms are similar to the previous dataset with an important di erence: did not remove the background noise after the reconstruction step, hence then the generated holograms will contain interference patterns from several noise sources like interference from other particles, sensor noise or dust particles in the background.
This limits the number of image transformations which could be applied and therefore reduces the amount of generated data. The ground truth of this set still contains the target objects without the noise background. It means the goal is to try to learn a model which is implicitly removes the noise during the hologram reconstruction. This dataset consists of 30100 hologram images. A few sample images are illustrated below in 4. The trained models were evaluated using several evaluation sets. The sets represent different levels of di culty. The list of evaluation dataset are as follows: { Training Set - This is the simplest set which includes the training sets, depending on the experiment. This set will always be similar to the training set. { Test set - This is the normal set from the training corpus, this is as usual a subset of the training set, which was not used for the model training or weight updates but only for testing. This set does not contain the same images as found in training, they will be still very similar because small steps in the propagation distance used during the hologram generation will lead to the very small di erence from one image to the next. { Blind test - This set contains the same training datasets but with di erent transformations e.g. rotations which were not part of the training. This means, still the same objects as used in the training model but the objects have di erent rotation. { Blind test of another dataset - These sets are even more di cult, it contains holograms of objects that are not part of the training corpus e.g. if the model was trained on the di raction corpus, then the evaluation was done on real di raction set.
They also contain di erent translations of objects. { Real holograms blind test - The most di cult evaluation data for the network will always be the real holograms test set. It contains the unaltered holograms as they are recorded by the in-line holography system. It is also important to note that this subset does not contain any holograms of the particles which were used for hologram generation. 5 5.1
Building Input Pipeline
The rst part of the experiment is to prepare the holograms for the input pipeline.
Handling this large amount of data is quite a challenging job. In general, the baseline
system, we used so far loading the arti cial holograms and preprocessing the holograms
was done with external libraries and then into the model with the feed dict method.
Tensor ow introduced the new version of Dataset API in an easy way to create a simple
record-oriented format which is popular for handling large data called TFRecords [
TFRecord is nothing but a binary le format or TensorFlow le. In a layman de nition, when to train a deep network, the input data has two ways to feed the network model. Loading the data with pure naive code and feed it into the computational graph which we used in our baseline system. The other way is to use an input pipeline that takes the list of lenames, shu es them, creates a le queue and decodes the data. This can be done with a TFRecord which was used in our current model. The advantage of using this type of le format is, it takes less space on disk, takes less time to copy and can be read from disk much more e ciently. 5.2
Model De nition
The experiments start with preparing the input pipeline for passing the data into the model. In the above gure, the right-hand side colored boxes represent the downsampling or contracting path and left-hand side colored boxes are upsampling or extracting path. These blocks are joined together with grey arrows marks are addressed by skip connections. The skip-connection is a technique that creates a shortcut to ignore the unnecessary next layers. The feature map is passed through the two types of connections: the continuous connection and the skip connection or shortcut. The adjacently previous layers as continuous connections and the connection that skips the layers are skip-connections. There is a ReLU activation function and batch normalization are succeeding the above each convolutional layer.
The feature maps are afterward fed to the next block which is similar to the rst block. The second block convolves and down-sampled from 32 feature maps with the size of 128x128 feature maps with the size of 64x64. Furthermore, the feature maps fed to the ve consecutive blocks with the same con guration of parameters, which are 256 kernels with the size of 3x3. The output of the seventh down-sampling block which is the last block of this part is 256 feature maps with a size of 4x4. The schematic representation of downsampling was shown in gure 6.
In the Up-sampling or extraction path, the Eighth block convolves and up-sampled from 256 feature maps of the 4x4 size to 256 feature maps with the size of 8x8. This block of layer contains the transposed convolution layer with 256 kernels with the size of 3x3 slicing 2 pixels for each step. The decoding block of layers contains the concatenation operation with the feature maps from the preceding layers through skip connections. After that, it combined feature maps followed by 3 regular convolutional layers with ReLU as activation function, batch normalization, and dropout. The schematic representation of upsampling was shown in the gure 7.
Furthermore, the feature maps are fed to the ve consecutive Up-sampling blocks with the same con guration of parameters, which are 32 kernels with a size of 3x3. The output of the nal block is a single feature map with the size of 256x256.
The model was trained on di cult datasets using an NVIDIA 12 GB Graphical Processing Unit. The default network settings are as follow: { Used Optimizer Adam { A learning rate of 0.0001 { image size of 256x256 pixels { Used training data is shu ed randomly The U-Net model was trained by Adam optimizer with an initial learning rate of 0.001. The model was learned with the number of epochs is 200 with the size of 64 training holograms. The criterion of evaluation is Sigmoid cross entropy.
The best results come from holograms with very small distance 0.0001 millimeter, while the worst outputs were generated by a large distance of 0.0600 millimeter. Di erent rotations didn't a ect the result. The loss curve of the training process is depicted in gure 8. The loss curve decreases over the training epochs via Sigmoid Cross Entropy Loss function.
Fig. 9: Comparing with objects loss curve Exp. Loss Function Id 1
Sigmoid Cross Entropy 2 3 4
Sigmoid Cross Entropy Mean Squared Error Mean Squared Error
Activation Function
Experimental Observation Linear Function The training and predicted results are
blurred Linear Function with the Images are blank subset of dataset Linear Function with the Training images are getting better but evalsubset of dataset uation images are still blank Linear Trained images are better with full set, still
validation images are not good
The rst experiments were trained using the baseline architecture without changing any parameters. The idea was, to see if the baseline model can be using the real holographic dataset. The di raction dataset, which continues to generated holograms based on real objects. After the training, the predicted results are blurred images. The reason for this might increase the complexity of data variations, which network was not able to learn. In the table 1 the list of each experiment were conducted with di erent hyperparameters and the results are not expected. 6.2
Improving generalization on unknown data After the model could be applied to the generated dataset, the next experiments aimed at improving performance on more di cult data, like rotation or object shapes not part of the training set. In other words, the goal was to test the generalization capabilities of the model. Therefore, in addition to the default network training settings di erent loss functions: Mean Pairwise Squared Error, Huber Loss and Mean Squared Error in combination with the activation functions: linear and sigmoid were tested. The maximum training time for each experiment was 48 hours. The experiments which were conducted during the period time were shown in the table 2, the rst column represents the experiment id, next columns are used loss function, data preprocessing methods, di erent datasets, output activation functions and results of each experiment.
Exp. Loss Function Id 5
Activation + Data Augmenta- Experimental Observations tion Mean Pairwise Sigmoid Activation Squared Error Huber Loss
Using more realistic training data The previous experiment showed a good performance of the model on generated data, for di erent levels of di culty. However, the model did not improve substantially for real holograms. Therefore, in the nal experiments, the real generated hologram dataset was used, which resembles a more realistic holograms. The goal was to check if this kind of data will help improving on real holograms. For this again di erent combinations of loss and activation functions were used. The list of conducted experiments with this dataset is shown in table 3. Experiment Loss Function Id 10 11 12 The Analysis of this experiment was impressed. The results were improved in terms of both training and evaluation. The loss curve is depicted in the gure 10. During the training corpus, the curve is slightly uctuating but in a decreasing order. Even the evaluation test which is a di erent level of di culty levels set results were shown the good to compare previous steps and leveled o with the training corpus curve. The blind test set of real holograms was showed better than previous and the curve goes in decreasing point but in times the curve rose when the steps are large.
Exp.Id- 11 Quantitative Analysis: The evaluation loss curve was leveled o with di erent blind datasets including the real holograms test set, in a small range of values. The loss curve decreases over the training steps were shown in gure 11. In the loss curves, the light blue color curve is the real hologram blind test set. It showed the best results among the previous experiments. It clearly showed that the experiment was a great success.
Exp.Id- 12 Quantitative Analysis: During the training corpus and evaluation, blind test sets were predicted great and decreasing enormously. In previous experiments the similar con guration without noise and augmentation operation was used, it predicts the blank images. But for this, it predicts good images due to the reason for applied noise and augmentation. We used this because to avoid the over tting problem. This quite an interesting nding for this experiment. The evaluation loss curves were showed similar to the previous one but not accurate. The curve uctuates when it reaches the large steps and predicts the over tting. 8
The proposed network architecture U-Net has proved to be a suitable approach for reconstructing real recorded holograms. The network design is highly adaptable for solving the image translation problems, especially recorded holograms, regardless of ground truth. During the experimental process, di erent loss functions were used because the architecture had faced problems with reconstructing the ground truth images, particularly for real holograms. In the end, the U-Net has shown the best performance using the Mean Pairwise Squared Error Loss Function combined with the Sigmoid Activation Functions.
In previous work, U-Net was successfully applied with several medical images, but not on generated images of holographic microscopy. We applied this network to reconstruct objects from real holograms and showed this is an additional way to utilize the benets of architecture. Furthermore, we modi ed the network by adding data augmentation and changing hyperparameters, which leads to an increase in its performance. The updated network was evaluated for several di culty levels of di erent test sets, to test the generalization capability. The entire holographic reconstruction process from the U-Net architecture was dependent on the four major steps outlined in this experimental setup: datasets, input pipeline, training and evaluating observations.