Characterization Method for Particle Extraction From Raw-Reconstructed Images Using U-Net

Digital holographic imaging can capture a volume of a particle field and reconstruct three-dimensional (3D) information of the volume from a two-dimensional (2D) hologram. However, it experiences a DC term, twin-images, defocus images of other particles and noise induced by the optical system. We propose the use of a U-net model to extract in-focus particles and encode the in-focus particles as squares at ground truth z. Meanwhile, zero-order images, twin-images, defocused images of other particle and noise induced by the optical system are filtered out. The central coordinate of the square represents the lateral position of the particle, and the side length of the square represents the particle diameter. The 2D raw-reconstructed images generated from the pre-processed hologram by utilizing backward Fresnel propagation serve as the input of the network. A dense block is designed and added to the encoder and decoder of the traditional U-net model. Each layer takes the inputs from all previous layers and passes the feature maps to all subsequent layers, thereby facilitating full characterization of the particles. The results show that the proposed U-net model can extract overlapping particles along the z-axis well, allowing the detection of dense particles. The use of that squares characterize particles makes it more convenient to obtain particle parameters.


INTRODUCTION
The particle fields comprise small objects, such as bubbles, biological cells, droplets. In recent years, 3D imaging has been widely used in particle detection (including shape, location, and motion) across many scientific domains, such as materials [1], chemical engineering [2][3][4], biology [5][6][7], medical sciences [8][9][10], and environmental science [11][12][13]. Digital holography (DH) encodes the 3D information of objects into a 2D hologram using the interference of the reference wave and object wave. Owning to only a single hologram can be reconstructed to restore the 3D information of the objects, DH has emerged as a powerful tool for 3D imaging in recent years. A spherical wave was first used as reference wave to observe particles distributed in water [14], the lateral position and size of each particle were gained. Thereafter, a plane wave was used as a reference wave to observe the bubbles in the air [15]. Conventional reconstruction methods have also been proposed. For example, the minimum intensity was applied to detect the edges of the bubbles from raw-reconstructed images. However, the minimum intensity method depends on the threshold setting to distinguish the particles from the background. Background noise and overlapped particles have serious effects [16]. Various criteria (such as edge sharpness and intensity distribution) [15,17,18] were applied to characterize the focus level of particles. However, these criteria are sensitive to the detailed characteristics and noise level in the holograms, limiting their application in low-concentration particle fields with low background and cross-interference noise. The deconvolution method [19,20] models the observed blur in 3D reconstruction as the convolution of an object and a point spread function (PSF). However, the PSF must be based on the known diffraction formula or obtained through a hologram of a point-like object in the experiment. The compressive holography method [21][22][23] is an effective reconstruction method to eliminate noise because of the sparsity of the signal, but it is time-consuming and requires complicated fine-tuning parameters to obtain optimal results.
Recently, machine learning using deep neural networks (DNNs) [24] has been applied to image analysis. First, the application of deep learning in DH appeared in medical examination [25][26][27][28]34] as well as in the classification of particles in holograms [29,30]. DNNs were also applied to acquire the depth information of particles [31,32], and autofocus is accomplished. Further studies have reported impressive results using DNNs for phase recovery [25,33], phase aberration compensation [34,35], hologram pixel superresolution [36,37] and digital holographic reconstruction [38][39][40]. Shimobaba et al. [41] first used a U-net model [42] to realize the holographic reconstruction of multiple particles. Shao et al. [43] proposed a U-net model to reconstruct hologram with higher-concentration particles. However, the sizes of the particles were not obtained. Li et al. [44] proposed using a short U-net with average pooling to extract in-focus particles at ground truth z and remove zero-order images, twin-images, and the defocused images of other particles from raw-reconstructed images; lateral position and the size of particles were obtained. However, it is complicated to obtain parameters of particles that are assumed to be circles [45]. Wu et al. [46] proposed the Dense-U-net and obtained particle information directly from the hologram and encoded them into a series of rectangles because it is convenient to identify and calculate the length and width of the rectangles. However, it is difficult to find completely overlapped particles along the z-axis from one single hologram. Inspired by the previous study, we found that encoding in-focus particles into squares is more conducive to the extraction of particle parameters. Simultaneously, to distinguish the particles completely overlapping along the z-axis, it will be better to train the neural network by feeding raw-reconstructed images that are generated from a hologram.
In this study, we propose the use of a U-net network to extract in-focus particles from raw-reconstructed images and encode them into a series of squares. The center coordinate of the square represents the lateral position of the particle, and the side length of the square represents the diameter of the particle. The rest of the paper is organized as follows: hologram preparation, U-net model, and characterization method are introduced in Principles. In Simulation Results the simulation results are presented. Experiment results are introduced in Experimental Results. The discussion is presented in Evaluation and Discussions. Finally, the conclusions are summarized in Conclusion.

PRINCIPLES Holography and Fresnel Diffraction Algorithm
Suppose there are n particles in the particle field in Figure 1A, each particle p i has a different size and different distance z i away from the camera. When a plane wave R(x, y) illuminates the particle field, the Fresnel diffraction [47] of each particle on the sensor plane is depicted in Eq. 1, and the object wave O(x, y), which is the coherent superposition of the diffraction fields by all the particles is shown in Eq. 2. The hologram is recorded on the sensor plane, which is the interference of R(x, y) and O(x, y). This process is depicted in Eq. 3.
Here (ξ, η), (x, y), (x′, y′), and (f x , f y ) represent the particle (lateral) plane, sensor plane, reconstructed plane, and spatial frequency domain, respectively. F{·} and F −1 {·} denote Fourier transform and inverse Fourier transform, respectively. I(x, y) denotes the recorded hologram, and O p (x, y) denotes the complex conjugate of O(x, y). The plane wave is assumed to be R(x, y) 1 in Eq. 3. Therefore, the pre-processed hologram is I final (x, y) I(x, y) − 1, as shown in Figure 1B. Moreover the pre-processed hologram is rewritten in Eq. 4, in which Figure 1C is obtained from the preprocessed hologram using 42, depicted in Eq. 5, at a certain ground truth z i . It comprises in-focus images, zero-order images, conjugate images, and defocus images of other particles [44]. Noted that the original objects are generated using the Mie scattering model [48].

Particle Characterization Method
Each particle in a particle field is characterized by (x, y, z i , D i ), where (x, y) represents the lateral position, z i denotes the distance to the sensor plane, and D i is the diameter of the particle. We propose the use of squares to represent particles. The central coordinate of the square represents the central coordinate of the particle, and the side length of the square is equal to the diameter of the particle. In this study, rawreconstructed images with in-focus particles, as shown in Figure 1C, serve as the input of the model. The areas encompassed by the red solid line in Figure 1C are in-focus particles. Via the special characterization method, the in-focus particles in Figure 1C are encoded into different squares, which is shown in Figure 1E. The images with squares shown in Figure 1E serve as the output of the model. And the U-net model is described below.

U-Net Model
A U-net model can be regarded as a black box. When many images are fed into it, the model learn the features of these images and obtain the mapping function f, which is calculated automatically through multi-layer network parameters, and is expressed as Eq. 6.
Where X, Y denote the input and output of the network, respectively, and f denotes the mapping function. θ denotes the parameters of the model, and it is updated every epoch during training. The proposed model, which is shown in Figure 1D, includes four down-samplings and four up-samplings. A dropout layer is applied to prevent overfitting. To simplify the training process, a residual neural network (ResNet) [49] is used to form a dense block. Each dense block consists of two Conv_Blocks which contains convolution, batch standardization, and activation layer. The output of each dense block is connected to the input of the dense block via a skip connection structure, which combines the high signal-to-noise ratio and the low signal-to-noise ratio features of the images for training in the deeper stages of the network. In this section, the rawreconstructed images serve as the input of the U-net model. The images shown in Figure 1E serve as output of the proposed model, meanwhile, mean square error is applied as the loss function of the model.

SIMULATION RESULTS
In the simulation section, we use the same Mie scattering model [48] to obtain the original particles. 51 holograms with the size of 256 × 256 are generated. A hologram shown in Figure 2A contains a volume 512 × 512 × 500 μm 3 . Thirteen transparent particles with diameter of 20, 30, and 40 μm are randomly distributed in this volume. The hologram is backward Fresnel propagated to 5 different distances, including 5,138, 5,238, 5,338, 5,438, and 5,538 μm, respectively, to generate the rawreconstructed images. Finally, 255 raw-reconstructed images corresponding to 51 holograms and 255 ground truth images are used as the dataset to train the U-net models. Because the particle field contains multiple transparent particles, we observe that the raw-reconstructed volume shown in Figure 2B is full of noise, including the zeros-order images, conjugate images, and defocused images of the other particles. The U-net model distinguishes in-focus particles from noisy images.
The hologram in Figure 2A is first used to test the model. The raw-reconstructed volume comprised five raw-reconstructed images is shown in Figure 2B. The corresponding predictions are presented in Figure 2C. We observe that the in-focus particles are extracted from the noisy images and encoded into squares, and the noise is filtered out. Hence, we obtain the coordinates and side lengths of the squares. The coordinates and the diameters are acquired, and the predicted location of particles are depicted in Figure 2E. The ground-truth distribution of the particle field is shown in Figure 2D. Figure 2F is the combination of ground truth and prediction; the coincidence of the blue circles and red dots represents the correct prediction, and the single blue circle represents the unpredicted particle. Figure 2G also shows a pre-processed hologram comprised of two overlapped particles along the z-axis, in which the spacing between the two particles equals the theoretical axial resolution of DH system (10 mm). The diameter of the particles equals 40 μm. The raw-reconstructed images of Figure 2G are depicted in Figure 2H. Figure 2I is the corresponding prediction of Figure 2H. We observe that the overlapped particles are successfully extracted at ground truth z. The theoretical axial resolution reaches 100 μm when a 10× microscope objectives (MO) is used. 75 raw-reconstructed images corresponding to 15 holograms with the same specifications but different particle distributions are generated for testing. The extraction rate of the particle field is 95.8%, and the lateral positioning error was less than 2 μm. The error of diameter is less than 4 μm. Simulation results show that the U-net network is successful in extracting the information of infocus particles from the raw-reconstructed images at ground truth z and encoding them into squares. Zero-order images, twin-images, and defocused images of other particles are filtered out simultaneously. When compared with [44], squares are more conducive to extracting particle parameters than circles. Compared with [46], this study can easily distinguish particles with the same lateral position when the depth spacing is greater than 100 μm. Frontiers in Physics | www.frontiersin.org February 2022 | Volume 9 | Article 816158 EXPERIMENTAL RESULTS Figure 3 shows the experimental setup, in which the wavelength of the laser is 632.8 nm, the size of the camera (charge-coupled device, CCD) is 2456 by 2058, and its pixel pitch is 3.45 μm. A hologram is generated by capturing a volume of 200 polystyrene particles per ml distributed in milli-Q water (diameter 90 ∼ 110 μm).
In the experiment, the holograms have different features from the holograms in simulation because they are composed of noise induced by the optical system (especially the laser). Therefore, a new model was trained using the dataset obtained in experiment. We use polystyrene spheres distributed in milli-Q water shown in Figure 3B to generate the first dataset (Supplementary Material), 60 holograms are obtained. And the second dataset containing 20 holograms is created using the method described below. The particles are deposited between every two adjacent slides in three slides, as shown in Figure 4D. The z-axis position of each slice is similar. Raw-reconstructed images in the dataset are generated by reversed holograms I revs (x, y) 255 − I(x, y) by utilizing BFP from 42 to 65 mm away from the sensor plane, and the corresponding ground truth images, as shown in Figures 4A,B,  respectively, are generated by the proposed method in [50] through image segmentation, and the minimum intensity is used as a focus metric. Noted that the method is not completely stable and accurate when the noise increases in a hologram which can be caused by cross interference [43] and oblique slides. Bad data (i.e., the holograms with severe noise mentioned above) is manually eliminated in this study. We adopt 120 rawreconstructed images (256 × 256) as the inputs in the dataset to train the U-net for 400 epochs, 20 raw-reconstructed images among them are from the second dataset which is generated by adhering particles between every two adjacent slides in three slides. The test result is carried out when the raw-reconstructed image in the second dataset, which is shown in Figure 4B, is predicted by the U-net model. The area encompassed by the turquoise solid lines is the in-focus particles. The corresponding prediction is shown in Figure 4C. We observe that the in-focused particles are successfully extracted and encoded into squares. Lateral position and diameter of each square are obtained. It is noted that the size of the rawreconstructed images for testing is 1024 × 1024 pixels, we divide the big raw-reconstructed image into smaller ones with a size 256 × 256 to conveniently feed them into the U-net model. Thereafter, they are stitched together to form the large images. Eventually, the test images can be rendered as a 3D volume depicted in Figure 4E.

EVALUATION AND DISCUSSIONS
As shown in Figure 4C, the in-focus particles in Figure 4B are properly restored, and the shapes of all the particles are squares, we can find squares to locate the lateral position of each particle, including the center coordinate and the diameter of each particle. As depicted in Figure 4E, the particle field is restored successfully. The positioning error is less than 3.45 μm (1 pixel) in the x, y directions, the error of diameter is less than 6.9 μm (2 pixel). The holograms are reconstructed when the depth spacing equals 200 μm. We use polystyrene spheres distributed in the milli-Q water to obtain one dataset, and we make the second dataset through adhering particles between every two adjacent slides in three slides because 1) the particles in the second dataset we make are few and scattered, which ensures the correctness of the lateral position of particles, 2) The thickness of the slides is known, so the depth spacing is certain, which is helpful to determine the position of particles along the z-axis. However, as the number of slides increases, the captured hologram is of poor quality. This may be primarily because the slides are not completely perpendicular to the optical axis of the optical system and noise is induced by the slides, therefore, we chose the simplest way by using three slides to divide the particles into two slices. Finally, the simulation and experimental results show that the proposed method is promising, and the time to obtain the prediction of a hologram is less than 96 s, which is much faster than the conventional iterative optimization methods. The rawreconstructed images are trained to match the ground truth images that in-focus particles are encoded into squares to easily obtain the information of the particles.

CONCLUSION
In this study, a U-net model is used to extract in-focus particles from the raw-reconstructed images at ground truth z and encode them into squares. Zero-order images, twin-images, defocused images of other particles and noise induced by the optical system are filtered out simultaneously. The center of the square represents the center of the particle, and the side length of the square represents the diameter of the particle. Finally, the information for each particle is obtained. We used squares instead of circles to represent particles because the squares are more convenient to acquire information of particles. The simulation and experimental results demonstrate that the proposed model is feasible. Eventually, a 3D particle field is restored.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.