Our experimental system is the differentiation induction process of human pluripotent stem cells into cardiomyocytes. The main differentiation process is shown in Figure 3, and images were captured and saved throughout the process.

First, the iPSC-18 cell line was chosen for induction experiments. iPSC-18 cells (Y00300, Takara) were routinely cultured in a PGM1 medium (CELLAPY) on growth factor–reduced Matrigel (corning)-coated 6-well plates. iPSC-18 cells were passaged every 4 days using EDTA (Gibco). hiPSCs were split into a CDM medium (Cauliscell Inc.) at a ratio of 1:10 before differentiation in 24-well or 96-well plates. When they attained ∼80–90% confluence, the medium was changed to a RPMI 1640 medium (RPMI, Gibco), 1x B27 without insulin (Gibco), and 100 U penicillin (Gibco), or RPMI+B27 minus, for shorting. During the first 48 h, hiPSCs were treated with CHIR99021 (CHIR, a WNT activator). From 48 to 72 h (day 3), the medium was changed to RPMI+B27 minus. During days 4–5, RPMI+B27 minus medium was supplemented with IWR1 (a WNT inhibitor). On day 6, IWR1 was withdrawn instead of the RPMI+B27 minus medium. After day 7 through to the end of the differentiation process (up to 14 days), the RPMI 1640 medium (RPMI, Gibco), 1 x B27 (Gibco), and 100 U penicillin (Gibco) were used and refreshed every 3 days.

After the final stage of induction (stage III: day 8–day 12), the cells were fixed in 4% paraformaldehyde (DING GUO) for 20 min at room temperature, permeabilized, and blocked in 3% normal donkey serum (Jackson) and 0.1% Triton X-100 for 45 min at room temperature. Then, the cells were incubated with a cTnT antibody (Thermo, MA5-12960, use 1:300) overnight at 4°C in PBS plus 0.1% Triton X-100 and 3% normal donkey serum (Jackson). The cells were washed with PBS and then incubated with secondary antibodies for 1 h at 37°C in a dark environment in PBS and 1% bovine serum albumin (BSA). Nuclei were stained with Hoechst 33342 (Yeasen) for 5 min at room temperature.

The irresistible degradation of image quality introduced in long-term live-cell experiments has been described previously. Errors are introduced during the culture medium changing operation on average every 24–48 h according to the experimental steps of the iPSC-CM differentiation system. An elaborate microscopy strategy must be carefully designed to reduce these errors as much as possible. A good microscopy strategy can maximize the image quality and speed up photographing so that higher throughput experimental image data can be obtained within one culture cycle. It is essential to find a balance between the imaging resolution ratio, experimental throughput, and imaging speed (Figure 2).

Because the imaging field of view of the microscope is limited to the optical device itself and light path design, multiple scanning imaging is required to expand the field of observation view, and the whole picture is stitched after the image acquisition. Therefore, the larger the observation field to be photographed, the more time-consuming it will be. It will cost more time to scan more culture chambers in a parallel multi-condition comparison observation of cell differentiation or reprogramming studies. On the other hand, it will further reduce the imaging speed of the system to perform the Z-axis layer imaging if the three-dimensional structure needs to be observed. Therefore, it is necessary to accelerate the imaging speed of each imaging experimental cycle to ensure the acceptable frequency of observation. If a higher imaging speed is required, a narrower imaging breadth is to be obtained, and vice versa. You cannot have your cake and eat it, too.

In addition, there are many options for different focusing strategies in an experiment. If the fixed focus or the one-time autofocus has been chosen, the out-of-focus caused by various emergencies in the long-term live-cell imaging process cannot be handled. For example, culture chamber expansion and contraction are caused by temperature or dead cell contamination. In particular, cells are raised into multiple layers because of cell growth. At this time, each layer of cells has its own focus surface due to the overlapping of multiple cell layers. Only multiple Z-axis microscopic imaging can obtain each sharp view of the overlapping cells. Notably, different cell types have different clonal heights, and the optimal focus surface may span more than 50–60 μM or even 100 μM in our iPSC-CM differentiation system.

On the other hand, if the autofocus every-time mode has been chosen, it is almost impossible to carry on the experiment because the focusing process will take plenty of time. Each well requires a 5 × 5 pattern mosaic tile stitching to obtain a square field of view of approximately 6.3 mm*6.3 mm according to the 96-well plates in this research. If only the center of each well is used as the focus reference point, then it will take approximately 48 min for 96 focus points to perform high-speed hardware autofocus. Moreover, if autofocus needs to be performed on each tile, the total time spent focusing will be 25 times larger, which is overtime to an incredible 20 h. Unfortunately, more focal points per well are required for the 24-well plates because of the larger culture area of each well. There is a way to combine a one-time autofocus strategy and multiple Z-axis imaging, then select the sharpest layer for use after imaging experiments (more images are shown in Supplementary Materials), but it also comes at the expense of time. These limits necessitate trade-offs between the imaging resolution ratio, experimental throughput, and imaging speed (Figure 2).

Here, the “Celldiscoverer 7,” a long-term live-cell culture instrument manufactured by Carl Zeiss, is used. It has an internal incubator to ensure regular cell growth, and the cell culture environment is kept stable at 37 °C with 5% CO₂. The ORCA-Flash 4.0 V3 digital CMOS camera is used as HD picture acquisition equipment. The effective resolution of the camera is 2,048*2,048 pixels. The objective is a ZEISS Plan-Apochromat ×5 objective. The objective can easily handle thin and thick vessel bottoms made of glass or plastic, which is essential to the hiPSC-CM differentiation system because our cells can only grow on plastic. With a ×2 tube lens, it achieves 10x/0.35 magnification and spatial resolution. Finally, the resolution ratio of all the photos is 0.65 μM per pixel.

For culture chambers, 96-well and 24-well plates produced by Falcon are used. A 2,048*2,048-pixel photo can cover a square of approximately 1.33*1.33 mm because of the resolution ratio of 0.65 μM per pixel. Therefore, the scanning imaging method was adopted for image acquisition, and the whole images were stitched after the imaging experiments. The larger the observation field is, the more mosaic tiles will be needed.

Multiple Z-axis layers are photographed to study the multiple Z-axis layer aggregation of the cells in the iPSC-CM differentiation process and to find the cause of blurring, and more importantly, to obtain the training data for the model. Eleven, seven, and five layers at 1.5 μM, 6 μM, and 18 μm intervals with total vertical distances of 15 μM, 36 μM, and 72 μM, respectively, were obtained for study (the images are shown in the Supplementary Materials).

Finally, the microscopic images were acquired by Carl Zeiss ZEN software version V2.5, and the images were saved in the CZI format or PNG format. A real-time microscopic image processing framework has been compiled for the long-term live-cell imaging system. It can automatically acquire and perform image preprocessing correspondingly, including image stitching and image segmentation. The segmented images will be sent to the CIEGAN for image enhancement for further analysis.

The deep convolutional generative adversarial network structure (Goodfellow et al., 2014; Radford et al., 2016) is adopted as the main body of the model to reconstruct high-quality and high-resolution images from low-quality microscopic cell images.

The GAN architecture in our model comprises a pair of generators and discriminators. Typically, the generator is trained to generate fake samples from random noise vector z. However, in our model, we take the blurred original image as the input z to enhance it. The input image flows through a pair of our carefully designed encoder-decoder-like structures in the generator. The latent implicit representation of the input image is obtained from the encoder module. The output image is precisely reconstructed using the information provided by the latent representation. On the other hand, the discriminator is trained to distinguish between the real cell images and the generated fake images. This framework can be represented as a two-player min-max game between generator G and discriminator D with value function V ( D , G ) : min G max D V ( D , G ) = Ε x ∼ p d a t a ( x ) [ log ( D ( x ) ) ] + Ε z ∼ p z ( z ) [ log ( 1 − D ( G ( z ) ) ) ] . (1) In Eq. 1, x represents the real-world high-resolution cell image examples. Discriminator D was trained to maximize the probability of assigning the correct label to both generated enhanced samples from G and the real-world cell image examples. At the same time, the generator G was trained to minimize log ( 1 − D ( G ( z ) ) ) simultaneously, that is, let the generated fake samples deceive the discriminator D to the maximum extent.

In the GAN structure, only the strongest generator survives in the game, which is very suitable for image restoration tasks. The adversarial loss ensures a high degree of realism of the image, making the image more natural and realistic. The following description will use x ^ to represent x ^ = G ( z ) , the generated image samples for brevity. The adversarial loss of the discriminator is formulated as Eq. 2: L a d v D = log ( D ( x ) ) + log ( 1 − D ( x ^ ) ) . (2)

The two parts are the true labels for the ground truth samples and the false labels for the generated samples. The optimization objective of the adversarial loss of the discriminator is formulated as Eq. 3: max D L a d v D = Ε x ∼ p d a t a [ log ( D ( x ) ) + log ( 1 − D ( x ^ ) ) ] . (3)

Similarly, the adversarial loss of the generator and its optimization objective are formulated as Eqs. 4, 5: L a d v G = log ( D ( x ^ ) ) , (4) min G L a d v G = Ε x ∼ p d a t a [ log ( D ( x ^ ) ) ] . (5)

In the GAN structure, the latent representation can capture valuable information in the input images, and the rest of the details and textures are handed over to network parameters for completion and reconstruction. However, it is not enough to determine the network parameters precisely in the generator only by the adversarial loss of the GAN, and more penalties are required to generate more accurate images and perform more refined image restoration.

Inspired by image style transfer (Isola et al., 2017), single-image super-resolution (SISR) methods (Yang et al., 2019; Ooi and Ibrahim, 2021), and high photorealistic image synthesis (Wang et al., 2018; 2021b), a series of image reconstruction losses are introduced to the model, such as pixel-wise loss and perceptual loss.

Specifically, using only reconstruction loss can reconstruct sharp images, but the generalization abilities are poor because of its pixel-wise properties. Therefore, images generated by reconstruction loss only may have excellent results superficially but suffer overfitting problems: just a single pixel translation may lead to model failure. Therefore, combining with the perceptual loss is a wise choice. The perceptual loss enables the contents and styles of the image reappearance. The reconstruction loss, also known as pixel-wise loss, is denoted as Eq. 6: L r e c = 1 B C H W ‖ x − x ^ ‖ 2 2 . (6)

In Eq. 6, B, C, H, and W represent the training batch size, the number of channels of the image or feature map, and the height and width of the feature map, respectively.

The perceptual loss comprises two parts: the feature loss part and the style loss part. The feature perceptual loss is formulated as Eq. 7 (Gatys et al., 2015; Johnson et al., 2016): L f e a t = ∑ i = 1 N 1 B C H W ‖ Φ i ( x ) − Φ i ( x ^ ) ‖ 2 2 . (7) In Eq. 7, Φ i is the i-th layer of a pre-trained VGG-16 or VGG-19 network (Simonyan and Zisserman, 2015), and Φ i ( x ) is the feature map of input image x. In the actual data flow, the shape of the feature map is the same as mentioned previously: B × C × H × W . N is the total number of VGG network layers. Here, we use the VGG-19 network, which is pre-trained on the ImageNet dataset (Deng et al., 2009). The style perceptual loss is formulated as Eq. 8: L s t y l e = ∑ i = 1 N 1 B C H W ‖ G r a m i ( x ) − G a r m i ( x ^ ) ‖ F 2 . (8) In Eq. 8, the Gram matrix can be calculated using the following formula: G r a m = A A T , where A represents a matrix. Here, the use of the squared Frobenius norm instead of the squared Euclidean distance was used before.

Nevertheless, a total variation regularization is imported to the model to remove noise and mosaics from images and further reduce the spikey artifacts of the generated images. The total variation regulator is formulated as Eq. 9: L t v = ∑ i , j H , W 1 B C H W ( ‖ x ^ i + 1 , j − x ^ i j ‖ 2 2 + ‖ x ^ i , j + 1 − x ^ i j ‖ 2 2 ) . (9)

In Eq. 9, x ^ i j represents a pixel from the generated enhanced image. Here, the rows and columns are calculated separately by the difference between adjacent pixels.

Finally, the loss of the CIEGAN is divided into two parts: the discriminator loss L D and the generator loss L G , shown as Eqs. 10, 11, respectively: L D = L a d v D , (10) L G = λ r e c L r e c + λ f e a t L f e a t + λ s t y l e L s t y l e + λ t v L t v + λ a d v G L a d v G . (11) The corresponding coefficients λ are added in front of different losses in the generator loss L G to balance the weights of different losses.

These losses and regularizations are merged together to reconstruct high-quality images and are referred to as the combined loss shown in Figure 4. The main structure of the CIEGAN model and the training and testing processes are depicted.

CIEGAN model coding mainly uses the TensorFlow deep learning framework (Abadi et al., 2016) and TF-slim library to build our generative adversarial nets. A cloud computing environment is used for model training and testing. The main hardware configuration list is an Intel Xeon Cascade Lake (2.5 GHz) 8-core processor, 32 GB of memory, and the CUDA computational acceleration unit is an NVIDIA T4 (with 15 GB of video memory).

Then, 128*128-pixel images, 256*256-pixel images, and 512*512-pixel images are successively tested on the CIEGAN model. According to the memory size of the CUDA unit, the network was trained using a batch size of 32, 20, and 4 images for the 128*128-pixel, 256*256-pixel, and 512*512-pixel inputs, respectively. Finally, the combination of 256*256-pixel image size and a batch size of 20 is chosen for the final training process according to the results.

The datasets used in training and testing come from two sources: 1) the original data are obtained from multiple Z-axis layers with an out-of-focus and sharp focus for each field of view. 2) Additional data were generated with a Gaussian blur from the original high-definition image to simulate the out-of-focus effect. In this way, more samples can be generated. Finally, a pair of blurry and high-resolution images of the same field of view are input into the model for training.

To ensure a stable and efficient training process and make the generator and discriminator converge, a multistep training strategy is adopted. 1) First, the generator is trained so that it can output primary-quality images. 2) Then, the discriminator is trained to identify fake images generated by the generator. 3) Finally, fire up the game and start the game process between the generator and the discriminator.

Initially, approximately 4,200 brightfield live-cell images (256*256 pixels) of the iPSC-CM differentiation process were used in the model test. The CIEGAN model only takes a few hours to achieve decent results on an NVIDIA T4. The time spent on training varies depending on the size of the training set in other applications. However, the use of the CIEGAN model is very fast; it can run 128 images (256*256 pixels) at a time of only a few minutes on VIDIA T4. Most of this is the load time of the model checkpoints. Once the model is loaded, its prediction timeliness is comparable to real-time processing. Nevertheless, our program provides automatic segmentation and assembly to handle larger input images. Finally, the performance of the CIEGAN should be similar to the same level of the CUDA computational acceleration unit. For example, it should have approximately the same time cost as this article on an NVIDIA 1080Ti GPU (12 GB).

First, the brightfield images in the hiPSC-CM differentiation process are enhanced for qualitative testing and the results are shown in Figure 5. A variety of cell morphologies are selected to test the robustness of this method. The thickness of the observed cells varies from flat monolayer to three-dimensional structures.

In the results, the traditional enhancement method (TE) is used to enhance the blurred input images for comparison. The TE method adopts the combination of the unsharp masking (Polesel et al., 2000; Deng, 2010) and contrast limited adaptive histogram equalization (CLAHE) (Pisano et al., 1998; Reza, 2004). Unsharp masking and CLAHE are classical tools for sharpness enhancement that can adaptively adjust and enhance the sharpness and contrast of the image, respectively. Here, the TE columns in Figure 5 show the results of the enhancement of the input cell images. Although the brightness and contrast of the images have been significantly improved through the enhancement of the traditional method, the blurring problem of the image has not been fundamentally solved. However, our CIEGAN method outperforms the traditional methods and benefit from the adversarial process and perceptual loss. The results of the CIEGAN are very close to the ground truth (GT), especially on the reconstruction of the fine structure of cells.

On the other hand, inspired by the Real-ESRGAN (Real Enhanced Super-Resolution Generative Adversarial Network) (Wang et al., 2021b), an improved CIEGAN was trained by using an enhanced version of the training sets, which we call CIEGAN plus (CIEGANP). The difference between them is the training inputs; the CIEGANP model is trained with enhanced ground truth images. Here, enhanced ground truth (EGT) is the result of image enhancement using unsharp masking and CLAHE methods for the GT. Interestingly, the image enhanced by the EGT method highlights the dead cells in the image (black dots) because the size of the dead cells or impurities is much smaller than the cells. Figure 5 (A–D) shows that the result of the CIEGANP is better than that of the CIEGAN in brightness and sharpness. Moreover, the results of the CIEGANP have less pepper noise or spikey artifacts than the enhanced ground truth (EGT) due to the introduced variation regulator.

To further evaluate the method, we performed a package of quantitative evaluations between traditional methods. The normalized root mean square error (NRMSE), peak signal-to-noise ratio (PSNR), and multi-scale structural similarity index (MS-SSIM) (Wang et al., 2003) are used in the image similarity assessment between the generated image and ground truth. The NRMSE reflects the pixel difference between the two images, and the smaller the value is, the better. The PSNR is the ratio of the maximum possible power of a signal to the power of corrupting noise that affects representation fidelity, which can objectively measure the image quality; the larger the value is, the better. The MS-SSIM is an improved version of the SSIM that is also used to measure image quality; the closer the value is to 1, the better. In addition, mutual information (MI) is used to measure the similarity of two images. The mutual information I between two pictures is formulated as Eq. 12: I ( X ; Y ) = ∑ y ∈ Y ∑ x ∈ X p ( x , y ) log ( p ( x , y ) p ( x ) p ( y ) ) . (12)

Mutual information in Eq. 12 describes the reciprocity between objects in two images, and the larger the value is, the higher the similarity between the two images will be.

We also used some no-reference methods for a single image quality evaluation in addition to the full reference method. The information gain and the estimation of the resolution ratio are used for evaluation after image enhancement. The information entropy H can be expressed as Eq. 13: H ( x ) = E ( I ( x ) ) = − ∑ x ∈ X p ( x ) log ( p ( x ) ) . (13) In Eq. 13, x represents the gray value of a pixel in the image, and p ( x ) is the probability of occurrence of its pixel gray value. For a normal 8-bit depth grayscale image, X contains the grayscale from 0–255 in this image. The information entropy is a nonnegative value, that is, it is used to describe the uncertainty of the pixels in the picture. The larger the information entropy is, the greater will be the amount of information contained in the picture.

On the other hand, resolution ratio estimation is widely used in biological image evaluation because it can indicate the actual resolution per pixel (Qiao et al., 2021). The resolution ratio calculation is performed by a decorrelation analysis, where the cross-correlation coefficient is expressed as Eq. 14 (Descloux et al., 2019): d ( r ) = ∫ R e { I ( k ) I n ( k ) M ( k , r ) } d k x d k y ∫ | I ( k ) | 2 d k x d k y ∫ | I n ( k ) M ( k , r ) | 2 d k x d k y . (14) In Eq. 14, k is the Fourier space coordinates, and I is the Fourier transform function. I ( k ) represents the Fourier transform of the input image, and I n ( k ) represents the normalization of I ( k ) . M ( k , r ) is the circular mask of the radius r (Descloux et al., 2019). The input image is passed through a series of high-pass filters and found the local maximum of the highest frequency; the normalized frequencies were denoted as k c . The resolution ratio is r e s o l u t i o n = 2 p k c , where p is the pixel size in photo acquisition.

The resolution ratio can measure the recognizability of structures in biological images; the smaller the resolution ratio value is, the greater the accuracy will be. The physical resolution in the iPSC-CM experiments is 0.65 μM per pixel, while the actual resolution of the ground truth images obtained may be poorer: 1.465 μM on average. The input images have a resolution of 4.305 μM on average due to out-of-focus, and our method can enhance this to 2.488 μM by the CIEGAN on average and 1.416 μM by the CIEGANP on average. Here, the traditional enhancement (TE) method only has a resolution of 3.546 μM on average. The results of the boxplot comparison are shown in Figure 5E.

An expert questionnaire is conducted to investigate whether the resulting pictures generated by this model are suitable for scientific research purposes. The results also show that the CIEGANP generally performs better than the other methods and has appropriate contrast and brightness.

The assessment of cardiomyocyte quality is required after the third stage of differentiation (Figure 3) in the iPSC-CM differentiation process. Therefore, fluorescent staining experiments were performed. Here, we stained two types of cell markers: cardiomyocyte-specific cTnT antibody and the nucleus-specific Hoechst 33342, which are used to assess the differentiation ratio and the cardiomyocyte quality. The enhancement results of the cell image stained with Hoechst 33342 are shown in Figure 6.

The CIEGAN algorithm has significantly enhanced the sharpness and contrast of the blurred cell staining images. It even obtained a higher signal-to-noise ratio than the ground truth. However, the performance in brightness is not perfect, which is a common problem of this model. The reason for this is that there are many black images with nothing in the training set. For this reason, the model trained with sharped ground-truth images as the CIEGAN plus has been introduced, and it significantly improves the sharpness and brightness of generating biological images while increasing the signal-to-noise ratio.

The enhanced cTnT fluorescent stained images are shown in Figure 7. The CIEGAN model achieves excellent generalization performance for different types of fluorescent images. The phototoxicity can be ignored in this experiment because the fluorescent staining method kills cells. Longer and stronger exposures can be used for imaging. However, photobleaching cannot be ignored because of the poor stability of dyes under strong light irradiation. It is not possible to use a strong intensity of light for a long time during the exposure process. Therefore, the balance between exposure time, light intensity, and image clarity also needs to be considered when taking photographs of fluorescence microscopy images. Nevertheless, it is sometimes impossible to obtain fluorescent photographs again because of severe photobleaching. In this case, the CIEGANP can not only deal with various out-of-focus images but can also enhance images with a low signal-to-noise ratio due to photobleaching.

The comparison with other methods is also carried out here. The “pix2pix” proposed by Isola et al. (2017) and the Real-ESRGAN proposed by Wang et al. (2021b) are used for comparison. The comparison results are shown in Figure 8.

First, the “pix2pix” model is trained on the dataset of brightfield images in the hiPSC-CM experiments and then achieves convergence. The results of “pix2pix” show its excellent performance, but there are a small number of artifacts and compression blur. On the other hand, the Real-ESRGAN shows its remarkable performance in real-world photos. Here, the Real-ESRGAN can increase the input biological image from 256*256 pixels to 1,024*1,024 pixels, which is 16 times the quantity of pixels. Because it was designed to perform super-resolution enhancement instead of dealing with blur degradations in biological micrographs, the high-definition pictures generated by the Real-ESRGAN are biologically distorted.