Abstract
Introduction: Although a powerful biological imaging technique, fluorescence lifetime imaging microscopy (FLIM) faces challenges such as a slow acquisition rate, a low signal-to-noise ratio (SNR), and high cost and complexity. To address the fundamental problem of low SNR in FLIM images, we demonstrate how to use pre-trained convolutional neural networks (CNNs) to reduce noise in FLIM measurements.
Methods: Our approach uses pre-learned models that have been previously validated on large datasets with different distributions than the training datasets, such as sample structures, noise distributions, and microscopy modalities in fluorescence microscopy, to eliminate the need to train a neural network from scratch or to acquire a large training dataset to denoise FLIM data. In addition, we are using the pre-trained networks in the inference stage, where the computation time is in milliseconds and accuracy is better than traditional denoising methods. To separate different fluorophores in lifetime images, the denoised images are then run through an unsupervised machine learning technique named “K-means clustering”.
Results and Discussion: The results of the experiments carried out on in vivo mouse kidney tissue, Bovine pulmonary artery endothelial (BPAE) fixed cells that have been fluorescently labeled, and mouse kidney fixed samples that have been fluorescently labeled show that our demonstrated method can effectively remove noise from FLIM images and improve segmentation accuracy. Additionally, the performance of our method on out-of-distribution highly scattering in vivo plant samples shows that it can also improve SNR in challenging imaging conditions. Our proposed method provides a fast and accurate way to segment fluorescence lifetime images captured using any FLIM system. It is especially effective for separating fluorophores in noisy FLIM images, which is common in in vivo imaging where averaging is not applicable. Our approach significantly improves the identification of vital biologically relevant structures in biomedical imaging applications.
1 Introduction
In addition to conventional fluorescence imaging, fluorescence lifetime imaging microscopy (FLIM) is a fundamental methodology in the biomedical imaging field that enhances the contrast of molecular structure. In light microscopy, FLIM is used to precisely measure the fluorescence decay lifetime of excited fluorophores (; ; ). By doing this, FLIM provides a highly effective tool for researchers. This important metric indicates the average time the fluorophore remains in the excitation condition before returning to the ground condition. Remarkably, FLIM offers distinct perspectives on many biochemical factors, such as ion concentrations, concentrations of dissolved gases, refractive index, pH levels, and micro-environmental conditions within living samples, immune to sample excitation power, fluorophore concentration (either adhesive or auto-fluorescence), and photobleaching, all of which are often very difficult to control in the majority of the experiments (). FLIM systems are broadly categorized into two types: time-domain FLIM (TD-FLIM) and frequency-domain FLIM (FD-FLIM). The TD-FLIM method involves the precise measurement of the time that elapses between the excitation of the sample by a pulsed laser and the photon arriving at the detector. In contrast, FD-FLIM exploits the modulation changes and the relative phase between the emitted fluorescence and the excitation using periodic modulated pulses for FLIM lifetime image generation.
Conventional FLIM systems (both TD-FLIM and FD-FLIM) are limited by slow processing speeds, a technical constraint; low signal-to-noise ratio (SNR), a fundamental constraint; and expensive and sophisticated hardware setup (). We have developed a new Instant FLIM system () that utilizes analog signal measurements for high-speed data collection, as illustrated in Figure 1, to overcome the technical limitations of the FLIM system. This approach eliminates bandwidth bottlenecks by incorporating pulse modulation with high-efficiency techniques and affordable deployment with readily available high-frequency devices such as mixers, low-pass filters, and phase shifters. Furthermore, the analog measurements of the down-converted frequency (intermediate frequency) signals used in our demonstrated frequency-domain Instant FLIM system enables the simultaneous measurement of fluorescence intensity, fluorescence lifetime, and frequency-domain phasors during 2-D, 3-D, or 4-D imaging for both live (in vivo), in vitro and fixed cell (ex vivo) applications.
FIGURE 1
Another fundamental problem is the low SNR, which remains to be addressed. Conventional image denoising techniques, such as averaging (mean filtering) within the same field-of-view (FOV) or median weight filtering, can enhance the SNR at the cost of image blurring and reduced frame rates with high computational time. Other image denoising method is block matching 3D filtering (BM3D) (
Machine learning (ML) has become more popular in recent years for its ability to enhance image processing performance, particularly in the area of image denoising (
2 Methods for fluorescence lifetime denoising
In this section, the traditional and machine learning methods used to perform lifetime image denoising are explored. The orthogonal axes of FLIM measurements (G and S measurements), as depicted in Figure 1, are utilized in our in-house developed instant FLIM system for the extraction of lifetime information through the difference between two complementary-phase mixers, presented in the subsequent equations.where, VIF(ϕ) indicates the intermediate frequency voltage at the given phase of ϕ.
Extracted lifetime is defined as, τ, as ratio of imaginary (S) to real (G) measurement at a given position using the equation τ = S/(ω ∗ G), ignoring the scaling factor. In the FD-FLIM system, the resulting lifetime is calculated using the following equationwhere ω = 2πfmod, and the laser pulse modulation frequency (fmod), is 80 MHz as implemented in our instant FLIM measurement setup. Phasor plots can be easier to interpret and visualize lifetime information in TD- and FD-FLIM measurements. This simplification is achieved by transforming the fluorescence lifetime value of each pixel into a point within a 2-D phasor plot. The x-coordinate is represented by the coordinate g, while the y-coordinate is represented by the coordinate s. Phasor coordinates (g and s pairs) are used to distinguish between different fluorophores and excited state reactions in a phasor plot. The phasors for the TD-FLIM time-correlated single photon count (TCSPC) system can be extracted using the transformations presented in the subsequent equations.where I(t) is the TCSPC fluorescence intensity information at the i − th pixel and time index of t. For the analysis of FD-FLIM system, the phasors are extracted through the calculation of the following equations.where ϕi and mi represent the phase shift and modulation degree change of the emission in comparison with the high-frequency excitation, respectively, at a random i − th pixel. The resulting phasor plot enables the identification of groups of pixels with similar fluorescence decays, which facilitates image segmentation. Therefore, the clustering method aids in identifying fluorophores with similar lifetime decay values on the phasor plot belonging to a single cluster.
2.1 Traditional methods
Typically, FLIM measurements are significantly noisy for in vivo imaging at low excitation power, and hence the phasor plot, potentially representing inaccurate clustering of sample boundaries to identify underneath fluorophores. One approach for phasor noise reduction in complex FLIM phasor axes is through conventional filtering methods like mean and median filters. However, these filters must be iteratively applied more than once to effectively reduce the noise and achieve a high SNR. Median filtering preserves edges in denoised image relative to mean filtering (
FIGURE 2

Using our custom-built instant FD-FLIM setup, the Noisy FLIM measurements were captured in an in vivo zebrafish embryo [Tg(sox10:megfp) at 2 days post-fertilization] in (A) and (B) for both G and S, while denoised images can be found in (D,E), respectively. Fluorescence intensity (C) and composite lifetime (F) images are provided, respectively. The composite lifetime corresponds to the hue saturation value (HSV) representation of the intensity and lifetime images combined, where the pixels’ brightness and hue are utilized to map the fluorescence intensity, and lifetimes, respectively. Excitation wavelength of 800 nm with a sample power of 5.0 mW and pixel dwell-time of 12 μs. To improve signal-to-noise (SNR), a 3D-volume of size 360 × 360 × 48, with each slice depth measuring 1 μm across 48 slices and a pixel dwell time of 12 μs, was averaged three times. FLIM Intensity image is normalized between 0 to 1 as shown in (C). Scale bar: 20 μm. Figure and data are derived from conference proceedings (
Phasor plots are obtained as a 2-D grid image in the XY plane, with real and imaginary FLIM measurement values projected onto the x − and y − axes, respectively. More details about phasor plots can be found in our lab’s previous review paper,
FIGURE 3

The raw phasor (A) and three iterations of the median filter [applied once (B), twice (C), and three times (D)] were applied to the complex FLIM measurements of G and S images of the in vivo zebrafish embryo Tg (sox10: megfp) at 2 days post-fertilization obtained using our custom-built instant FD-FLIM setup, respectively. In addition, for raw FLIM measurements refer to Figure 2, which includes both intensity and composite lifetime information. Figure and data are derived from conference proceedings (
TABLE 1
| 3-D volume name | Execution time average ± standard error (seconds) | |
|---|---|---|
| Median filter | G_volume | 0.276 ± 0.0027 (once) |
| 0.355 ± 0.0022 (twice) | ||
| 0.432 ± 0.0025 (three times) | ||
| S_volume | 0.281 ± 0.0026 (once) | |
| 0.347 ± 0.0021 (twice) | ||
| 0.423 ± 0.0024 (three times) |
Computation time for median filtering on FLIM measurement data of real G and imaginary S components of entire 3D volume stack in MATLAB (
2.2 Pre-trained ML models
In this section, we demonstrate a unique pretrained CNN model trained on a diverse set of fluorescence microscopy intensity images utilizing CNNs for instantaneous denoising of both the orthogonal dimensions of the complex phasor axes (G, and S images) through the use of the FIJI tool, an image processing software (
In our research, we utilized two pre-trained machine learning models specifically designed for denoising fluorescence images (G and S of complex phasor components) in the presence of mixed Poisson-Gaussian (MPG) noise. The first model, known as the “DnCNN,” is a supervised deep convolutional neural network (CNN) constructed using the DnCNN architecture (
Table 2 shows summary of the samples collected to demonstrate our approach using different FLIM systems and different samples to generate fluorescence microscopy denoised images and denoised phasor images. To use pre-trained ML models for images from a different distribution than the training dataset, it is important to analyze the noise distribution of the input images (G and S). Figure 1 shows the extraction of the complex phasor orthogonal axes from the FLIM measurement using the difference between two complementary phases: G = VIF(0.5π) − VIF(1.5π) and S = VIF(0) − VIF(π). The FLIM measurement signal, VIF(ϕ) originates from a photomultiplier tube (PMT) and other analog device components including low-pass filters (LPFs), which results in a noise distribution that contains both Poisson and Gaussian noise, also known as mixed Poisson-Gaussian (MPG) noise. The real (G) and imaginary (S) images of the FLIM measurement follow a Skellam distribution, which is the difference between two Poisson distributions (
TABLE 2
| Sl. No | Sample name | Imaging type | Fluorophores labeled | Sample power (mW) | FLIM setup |
|---|---|---|---|---|---|
| 1 | Zebrafish | in vivo | EGFP | 5 | Instant FLIM |
| 2 | Mouse Kidney | in vivo | Auto fluorescence | 5 | Commercial FLIM |
| 3 | BPAE cells | ex vivo | DAPI, Alexa Fluor 488 phalloidin, MitoTracker Red CMXRos | 5 | Instant FLIM |
| 4 | Mouse Kidney | ex vivo | DAPI, Alexa Fluor 488 wheat germ agglutinin, Alexa Fluor 568 phalloidin | 3.3 | Instant FLIM |
| 5 | Plant images | in vivo | Auto fluorescence | 3 & 5 | Instant FLIM |
List of the samples experimentally captured to demonstrated our denoising approach along with experimental conditions for the reproduction of the sample images. Please note that for all of these samples are imaged using two-photon FD-FLIM systems with the two-photon excitation wavelength of 800 nm.
Figure 4 depicts the proposed workflow for utilizing pre-trained ‘DnCNN’ machine learning models to denoise FLIM measurements and extract the phasor from the resultant denoised complex FLIM measurements. The method includes pre-processing and post-processing steps to restrict the range of orthogonal phasor axes to use the full-dynamic range of demonstrated pre-trained image denoising models and convert them back to their original scale during the inference process. Figures 2D, E display the denoised images of G and S, respectively, for one imaging plane in the 3D volume stack. Additionally, Figure 4 illustrates noisy phasors and denoised phasors through the use of our pre-trained DnCNN ML model. Figure 4.
FIGURE 4

A block diagram illustrating the proposed methodology applied to the 3D volume stack depicted in Figure 2. The phasor representation before (shown in red arrow) and after (shown in green arrow) FLIM measurement denoising (noisy and denoised phasors) using our pre-trained DnCNN denoising model. Figure and data are derived from conference proceedings (
To quantitatively assess the performance of lifetime image denoising methods, we employ the phasor representation of the fluorophores (
While quantitative metrics like PSNR and SSIM cannot be calculated due to the lack of an averaged image, the phasor approach provides a qualitative assessment of the denoising effectiveness of pre-trained CNNs in in vivo zebrafish embryos FLIM measurements.
Finally, the segmentation in FLIM measurements that represents different fluorophores with accurate lifetime values (dividing into segments) is crucial for identifying fluorophores location of various lifetime values. The location of the fluorophores and their lifetime values are unknown in intravital imaging, where autofluorescence dominates. In a phasor plot, phasors with similar fluorescence decays cluster together, resulting in the need for accurate segmentation. Segmentation of the phasor occurs by selecting a cluster representing fluorophores with similar decay rates or lifetimes and assigning distinct colors to these clusters. The resulting images display the respective fluorophores using each color. To perform segmentation, typically a region marked with different colors is selected by the user in the phasor, and lifetime information is used to identify the fluorophore that belongs to this region. However, this method is prone to time consumption and unreliable outcomes, is contingent upon the user’s region selection, and cannot be reproduced. To prevent biased segmentation results, we suggest a fresh, impartial process for automatic phasor labeling, utilizing the unsupervised method named “K-means clustering” (
3 Results and discussion
We applied pre-trained ML models to denoise FLIM data, resulting in clean phasors. These denoised phasors were further processed using K-means clustering across various test samples, as shown in the below sub-sections.
3.1 In vivo samples with phasor-segmentation
Figure 5 To illustrate our proposed pre-trained ML models for the FLIM lifetime image denoising, an in vivo mouse kidney sample is chosen and captured under a commercial FLIM setup. Figures 5A, B illustrate the fluorescence intensity and lifetime, respectively, for an in vivo mouse kidney (sample taken from The Jackson Laboratory, a male variant C57BL/6J mice of age at 8–10 weeks) acquired using a commercial FD-FLIM digital system (
FIGURE 5

Application of K-means clustering segmentation on in vivo mouse kidney phasor data obtained through a commercial FD-FLIM system (
To resolve the problem of noisy FLIM measurements, we performed the pre-trained DnCNN model to denoise the FLIM measurements (complex phasor plane). This improves the phasor SNR, which is demonstrated in Figure 5D, exhibiting a high SNR compared to the noisy phasor as shown in Figure 5C. Once the denoised phasor is extracted, the K-means clustering method is used to achieve precise segments for the two proximal tubules (S1/DT and S2), as displayed in Figure 5F. After segmentation and denoising, the overlapped segmentation image was generated and can be found in Figure 5H. The red and blue tubules displayed in the image correspond to the S1/DT proximal tubules combined as one cluster and the S2 proximal tubules as another cluster, respectively. Hence, the demonstrated pre-trained DnCNN model provides rapid and precise automated segmented clusters of fluorescence lifetime images obtained through in vivo FLIM measurements. In cases where FLIM measurements exhibit noise, pre-trained CNN-based FLIM phasor denoising and segmentation using K-means clustering methods can prove beneficial. Clustering has been found to be an effective approach for improving biological structure detection in biomedical imaging research applications. An example where our method can be used to accurately identify microtubules of an in vivo mouse kidney sample is shown in Figure 5H. The absence of an averaged image hinders the calculation of quantitative metrics like PSNR and SSIM, mirroring the situation in zebrafish embryo FLIM measurements. Despite this limitation, the phasor approach offers a valuable qualitative evaluation of the denoising efficacy of pre-trained CNNs in the context of in vivo mouse kidney FLIM measurements.
3.2 Phasor denoising and clustering in fixed fluorescence samples
In addition to the in vivo mouse kidney samples, we show our demonstrated pre-trained ML model for FLIM denoising on a fixed BPAE sample. We acquired noisy BPAE sample images (Invitrogen prepared slide #1 F36924 by FluoCells) using our in-house customized two-photon frequency-domain fluorescence lifetime microscopy system, as described in
The qualitative outcomes obtained from the fixed BPAE sample are illustrated in Figure 6A, the noisy intensity of the fixed BPAE sample cell, acquired experimentally via the two-photon Instant FLIM system, is depicted (presented as a single-channel image using Cyan hot false color). Figures 6B–D display a composite lifetime image derived from the noisy FLIM data, denoised from the noisy FLIM BPAE data using a pre-trained DnCNN model, and averaged measurements (averaged lifetime over 5 acquisitions within the same FOV) representing high-SNR images for reference, respectively. Figures 6E–H show the selected ROIs for the intensity, composite lifetime images of noisy, DnCNN-denoised, and averaged lifetime images, respectively. Denoised lifetime images exhibit notably fewer red pixels, indicating reduced noise and accurate lifetime values compared to noisy lifetime images. The phasors of the selected ROI in the noisy, denoised, and averaged FLIM measurements are presented in Figures 6I–K, respectively Figure 6. Clear identification of the nucleus and mitochondria is evident in the denoised lifetime and phasor images in comparison with the noisy lifetime. The denoised images closely align with the averaged image and exhibit improved SNR values. In Figures 6A–D, the upper series signifies complete FOV of size 512 × 512, while the lower series in Figures 6E–H represents ROI marked by the yellow square of size 125 × 125 of the corresponding upper series images. The results of K-means clustering on the phasors of the noisy, denoised, and averaged FLIM measurements are presented in Figures 6L–N, respectively. The PSNR values of the BPAE samples lifetime images noisy and denoised using DnCNN pre-trained CNNs methods are 13.69 dB and 18.07 dB, respectively. Similarly, the SSIM values of the lifetime images noisy and denoised using DnCNN methods are 0.076 and 0.173, respectively. From the PSNR values, there is an improvement in image quality of 4.38 dB due to DnCNN pre-trained CNNs denoising method. In a previous study, we demonstrated the superior performance of our pre-trained CNNs compared to established image denoising methods like BM3D (
FIGURE 6

Phasor representation was obtained for a fixed BPAE sample using our instant FD-FLIM system, and pre-trained ML models were applied before and after. The FLIM data was acquired following (
We also conducted imaging on another fixed mouse kidney sample [prepared slide #3 (F-24630) by FluoCells], where mouse kidney samples were stained with DAPI (nuclei using blue-fluorescent DNA stain), Alexa Fluor 488 wheat germ agglutinin (highlighting elements of the glomeruli and convoluted tubule using green-fluorescent lectin), and Alexa Fluor 568 phalloidin (abundant in glomeruli and the brush border using red-fluorescent filamentous actin) utilizing our Instant FLIM system (
Figure 7A presents the noisy image of the fixed mouse kidney sample, captured through our specialized two-photon FLIM system, displayed as a single-channel image with magenta hot false color. Concurrently, Figures 7B–D demonstrate the composite lifetime of noisy FLIM data, the denoising effect of the pre-trained ML model on the noisy FLIM mouse kidney fixed sample data, and averaged lifetime measurements (averaged over 5 samples within the same FOV), providing high-SNR images for comparison, respectively. Furthermore, Figures 7E–H present the ROIs for the intensity, composite lifetime images of noisy, DnCNN-denoised, and averaged lifetime images, respectively. Notably, denoised lifetime images contain fewer red pixels, indicating diminished noise and precise lifetime values compared to noisy counterparts. Phasors for the noisy, denoised, and averaged FLIM measurements are displayed in Figures 7I–K, respectively. The results of K-means clustering on the phasors of the noisy, denoised, and averaged FLIM measurements are depicted in Figures 7L–N, respectively, illuminating the effectiveness of the denoising process. The PSNR values of the fixed mouse kidney noisy and denoised using DnCNN methods lifetime images are 17.63 dB and 22.10 dB, respectively. Similarly, the SSIM values of the noisy and denoised using DnCNN methods lifetime images are 0.190 and 0.355, respectively. From the PSNR values, there is an improvement in image quality of 4.47 dB due to DnCNN pre-trained CNNs denoising method. Figure 7.
FIGURE 7

Phasor representation of a mouse kidney fixed sample was analyzed before and after application of pre-trained ML models and the FLIM images were acquired using our instant FD-FLIM system (
3.3 Out-of-distribution samples: high-scattered plant tissues
In our research, deep learning models proficient in training datasets acquired from FLIM systems depict information as 2D sections per channel in the 3D volume across specific time frames. Addressing the challenge of model performance on data that differs from the training set is crucial, especially for complex structures like highly scattering plant tissues. The intricate cellulose cell wall structures in plants cause high optical scattering coefficients, making depth-resolved imaging challenging for conventional microscopy systems. Confocal laser scanning microscopy (CLSM) provides limited depth resolution due to scattering, whereas multi-photon microscopy (MPM) offers deeper penetration. Given this, our approach’s validation on plant samples that differ from the training set is crucial, demonstrating its ability to adapt to new data and generalize the pre-trained models. To assess our pre-trained ML models for lifetime denoising, we used our instant FLIM system (
FIGURE 8

FLIM images of the in vivo bean plant spongy mesophyll layer: intensity image (A), lifetime images where raw lifetime (B), median filter denoised lifetime (C), pre-trained DnCNN ML model denoised lifetime (D), and ground-truth (averaged) lifetime image (E), respectively. Likewise, the designated ROI, indicated within the yellow box, is depicted in panels (F–I) to display the corresponding selected field-of-view. FLIM intensity and lifetime images are presented in false colors, where each image is a gray-scale image. Excitation wavelength: 800 nm; sample power: 3 mW; pixel dwell-time: 12 μs. FLIM Intensity image is normalized between 0 to 1 as shown in (A). Scale bar, 5 μm.
Bean plant leaves (grown in our laboratory at room temperature from Ferry-Morse bean seeds) are imaged using our custom-built MPM-FLIM [InstantFLIM (
FIGURE 9

FLIM images of the in vivo bean plant spongy mesophyll layer: lifetime image distribution with Gaussian fit for the raw image (A), median filter denoised image (B), DnCNN denoised lifetime using our ImageJ plugin (C), and averaged lifetime image (D), respectively. From the denoised image and averaged images, two-lifetime distributions can be identified, which are missing in the case of the raw lifetime image distribution.
Similarly, Figure 10 shows the upper epidermal layer of the bean plant is imaged using our InstantFLIM system (
FIGURE 10

FLIM images of the in vivo bean plant Upper epidermis layer: intensity image (A), lifetime images where raw lifetime (B), median filter denoised lifetime (C), DnCNN denoised lifetime using our ImageJ plugin (D), and averaged lifetime image (E), respectively. Likewise, the designated ROI, indicated within the yellow box, is depicted in panels (F–I) to display the corresponding selected field-of-view. FLIM intensity and lifetime images are presented in false colors, where each image is a gray-scale image. Excitation wavelength: 800 nm; power: 5 mW; pixel dwell-time: 12 μs. FLIM Intensity image is normalized between 0 to 1 as shown in (C). Scale bar, 5 μm.
Once the phasor axes (G and S images) of complex FLIM measurement data are denoised using the pre-trained DnCNN model, the denoised phasor is extracted, as shown in Figure 11. In addition, the phasor segments of the phasor are extracted using the K-means clustering method. Figure 11A shows the single-channel gray-scale image in false magenta color intensity of the in vivo bean plant leaf spongy mesophyll layer. Figure 11B–D show the single channel gray-scale image in false cyan color raw, DnCNN denoised, and averaged lifetime images, respectively. Figure 11E–G show the false-color composite HSV lifetime image, where value (pixel brightness) represents fluorescence intensity and hue (color) represents fluorescence lifetime of the raw, DnCNN-denoised, and averaged lifetime images, respectively. Figure 11H–J show the phasors of the raw, DnCNN denoised, and averaged phasors, respectively. Figure 11H shows the phasor with a larger noise distribution, while the denoised phasor in Figure 11I shows reduced noise with improved SNR. In addition, the phasors are divided into two groups to represent the chlorophyll and cytosolic structures, as shown in red and blue colors, respectively. Figure 11K–M show clusters of the two fluorophores in the raw, DnCNN-denoised, and averaged lifetime clusters, where the blue color pixels represent the chlorophyll and red color pixels represent the cytosolic structures. Figure 11.
FIGURE 11

FLIM images of the in vivo bean plant spongy mesophyll layer: intensity image (A), lifetime images where raw lifetime (B), DnCNN denoised lifetime using our ImageJ plugin (C), and averaged lifetime image (D), respectively. FLIM intensity and lifetime images are presented in false colors, where each image is a gray-scale image. Composite lifetime images are shown in (E–G), respectively, in the HSV format where value (pixel brightness) is mapped as fluorescence intensity and hue (color) is mapped as fluorescence lifetime. Phasor diagrams with K-means clustering with 2 clusters (k = 2) are shown in (H–J), respectively. Overlap clusters of the lifetime images are shown in (K–M), respectively. FLIM Intensity image is normalized between 0 to 1 as shown in (A). Excitation wavelength: 800 nm; power: 3 mW; pixel dwell-time: 12 μs. Scale bar, 5 μm.
Finally, FLIM lifetime denoising results using the Noise2Noise pre-trained CNN model are provided in GitHub4. The outcomes disclosed in this study are available publicly and can be accessed through the same GitHub repository.
3.4 Limitation of pre-trained CNNs on lifetime image denoising
Our pre-trained ML models consistently perform well on unseen fluorescence intensity image structures and samples, demonstrating their ability to generalize and avoid over-fitting. This means they can effectively denoise fluorescence lifetime images with various structures and samples (such as fluorescence nanobeads, etc.,) that differ from those during training. However, when it comes to noise distribution, denoising lifetime images using pre-trained CNNs trained on fluorescence intensity images can be challenging due to the complex noise distribution of fluorescence lifetime images. Our method addresses this challenge by employing the complex phasor representation of FLIM images, where each component of complex FLIM measurement provides a noise distribution of the Skellam distribution that can be approximated to Gaussian noise and is compatible with pre-trained CNNs noise distribution of MPG noise.
Recent FLIM image acquisition methods often produce multidimensional data (spatial (XYZ), temporal (T), and multiple channels at different emission filters (λ1, λ2), which are typically presented in 2D sectional images of complex FLIM measurement. The noise in each 2D sectional image follows the Skellam distribution. Further performance improvements could be achieved by training models directly on these 2D images. This would require a large training dataset, which can be difficult and expensive to acquire, especially for in vivo imaging. However, this approach has the potential to significantly improve fluorescence lifetime phasor denoising. Currently, our approach is limited to processing 3D sections of FLIM data, meaning that multidimensional FLIM data requires pre-processing into 3D sections of XYZ (entire volume at each discrete time interval) or XYT (3D stack of 2D planes at different time intervals) in each acquisition channel before applying our method. Following the application of our image denoising method, the denoised results must be combined as part of a post-processing step. Finally, it is always recommended to check if the generated denoised lifetime images have any artifacts using the existing quantitative metrics such as the phasor position of the fluorophores to indicate accurate lifetime values.
4 Conclusion
To overcome the challenges of slow data capture (longer time for imaging), low SNR, and costly setups in fluorescence lifetime imaging microscopy (FLIM), we introduce Instant FLIM, a unique, rapid processing FLIM instrument using real-time signal processing and commercial analog processing devices to provide fluorescence intensity, lifetime, and phasors for multi-dimensional FLIM data, including 2-D (XY plane), 3-D (XYZ with Z: depth), or 4-D (XYZT with Z: depth, T: time) in vivo imaging. This paper demonstrated the utilization of pre-trained deep learning-based image denoising models to remove noise from complex FLIM measurements before extracting the sample fluorescence lifetime and sample phasor plot. The denoised phasor shows a qualitative improvement in SNR in FLIM images. Subsequent application of denoised phasor data to K-means clustering segmentation reveals distinct segments corresponding to different fluorophores. Our demonstrated method has been rigorously tested on diverse samples, including in vivo mouse kidney tissue, fixed BPAE samples, fixed mouse kidney samples, and highly scattering plant samples, demonstrating its efficacy in enhancing the detection of key biological structures in FLIM applications. Overall, the combination of Instant FLIM and pre-trained ML models for denoising offers a fast and accurate solution for fluorescence image denoising, yielding high SNR and accurate segmentation.
Statements
Data availability statement
The original contributions presented in the study are included in the article are available in the GITHub location: https://github.com/ND-HowardGroup/FLIM_Denoising_using_Pretrained_CNNs, further inquiries can be directed to the corresponding author.
Author contributions
VM: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing–original draft, Writing–review and editing. JB: Investigation, Writing–review and editing. CS: Investigation, Writing–review and editing. XY: Investigation, Writing–review and editing. SH: Funding acquisition, Supervision, Writing–review and editing.
Funding
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This research has been made possible through funding from the National Science Foundation (NSF) under Grant No. CBET-1554516 and more details are provided here https://www.nsf.gov/awardsearch/showAward?AWD_ID=1554516.
Acknowledgments
The authors would like to acknowledge the valuable contributions of Takashi Hato, Pierre C. Dagher, and Prof. Kenneth W. Dunn from the Division of Nephrology, Indiana University, for providing the in vivo mouse kidney sample FLIM measurements captured using a commercial FLIM system. Please note that part of the datasets and analysis are provided in our previous conference paper (
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
1.^https://github.com/ND-HowardGroup/Instant-Image-Denoising/tree/master/Plugins/Image_Denoising_Plugins_Journal/Plugin_Targets
2.^https://www.thermofisher.com/order/catalog/product/F36924
3.^https://www.thermofisher.com/order/catalog/product/F24630
4.^https://github.com/ND-HowardGroup/FLIM_Denoising_using_Pretrained_CNNs.git
References
1
ChangC.-W.SudD.MycekM.-A. (2007). Fluorescence lifetime imaging microscopy. Methods Cell Biol.81, 495–524. 10.1016/s0091-679x(06)81024-1
2
contributorsW. (2004). Skellam distribution. Available at: https://en.wikipedia.org/wiki/Skellam_distribution.
3
DabovK.FoiA.KatkovnikV.EgiazarianK. (2007). Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process.16, 2080–2095. 10.1109/tip.2007.901238
4
DattaR.HeasterT. M.SharickJ. T.GilletteA. A.SkalaM. C. (2020). Fluorescence lifetime imaging microscopy: fundamentals and advances in instrumentation, analysis, and applications. J. Biomed. Opt.25, 071203. 10.1117/1.jbo.25.7.071203
5
DigmanM. A.GrattonE.MarcuL.FrenchP.ElsonD. (2014). The phasor approach to fluorescence lifetime imaging: exploiting phasor linear properties. Fluoresc. Lifetime Spectrosc. Imaging, 235–248. 10.1201/b17018-14
6
GabrielP. R. (2008). SSIM: a java plugin in ImageJ. Available at: https://imagej.nih.gov/ij/plugins/ssim-index.html.
7
GoodfellowI.BengioY.CourvilleA. (2016). Deep learning. MIT press.
8
GriffinT. F. (1992). Distribution of the ratio of two Poisson random variables. Available at: https://ttu-ir.tdl.org/bitstream/handle/2346/59954/31295007034522.pdf.
9
HoreA.ZiouD. (2010). “Image quality metrics: PSNR vs. SSIM,” in 2010 20th international conference on pattern recognition (IEEE), 2366–2369.
10
JainA. K. (2010). Data clustering: 50 years beyond K-means. Pattern Recognit. Lett.31, 651–666. 10.1016/j.patrec.2009.09.011
11
LehtinenJ.MunkbergJ.HasselgrenJ.LaineS.KarrasT.AittalaM.et al (2018). Noise2Noise: learning image restoration without clean data. Proc. Mach. Learn. Res. (PMLR)80, 2965–2974.
12
MacQueenJ.et al (1967). “Some methods for classification and analysis of multivariate observations,” in Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, Oakland, CA, USA1 (14), 281–297.
13
MannamV. (2022). Overcoming fundamental limits of three-dimensional in vivo fluorescence imaging using machine learning. 10.7274/5x21td99n58
14
MannamV.HowardS. (2023). Small training dataset convolutional neural networks for application-specific super-resolution microscopy. J. Biomed. Opt.28, 036501. 10.1117/1.jbo.28.3.036501
15
MannamV.KazemiA. (2020). Performance analysis of semi-supervised learning in the small-data regime using VAEs. arXiv preprint arXiv:2002.12164.
16
MannamV.ZhangY.YuanX.HatoT.DagherP. C.NicholsE. L.et al (2021). Convolutional neural network denoising in fluorescence lifetime imaging microscopy (FLIM). In Multiphoton Microscopy in the Biomedical Sciences XXI (International Society for Optics and Photonics (SPIE Photonics West)), vol. 11648, 116481C
17
MannamV.ZhangY.YuanX.RavasioC.HowardS. S. (2020a). Machine learning for faster and smarter fluorescence lifetime imaging microscopy. J. Phys. Photonics2, 042005. 10.1088/2515-7647/abac1a
18
MannamV.ZhangY.ZhuY.HowardS. (2020b). Instant image denoising plugin for ImageJ using convolutional neural networks Microscopy Histopathology and Analytics (Optical Society of America). MW2A–3.
19
MannamV.ZhangY.ZhuY.NicholsE.WangQ.SundaresanV.et al (2022). Real-time image denoising of mixed Poisson–Gaussian noise in fluorescence microscopy images using ImageJ. Optica9, 335–345. 10.1364/optica.448287
20
MATLAB (2019). 9.7.0.1190202 (R2019b) (natick, Massachusetts: the MathWorks inc.).
21
NehmeE.WeissL. E.MichaeliT.ShechtmanY. (2018). Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica5, 458–464. 10.1364/optica.5.000458
22
RuedenC.SchmidtD.WilhelmB. (2017). ImageJ tensorflow library. Available at: https://github.com/imagej/imagej-tensorflow.
23
SageD. (2017). ImageJ’s plugin to assess the quality of images. Available at: http://bigwww.epfl.ch/sage/soft/snr/.
24
von ChamierL.LaineR. F.JukkalaJ.SpahnC.KrentzelD.NehmeE.et al (2021). Democratising deep learning for microscopy with ZeroCostDL4Mic. Nat. Commun.12, 2276–2293. 10.1038/s41467-021-22518-0
25
WangP.HechtF.OssatoG.TilleS.FraserS.JungeJ. (2021). Complex wavelet filter improves FLIM phasors for photon starved imaging experiments. Biomed. Opt. Express12, 3463–3473. 10.1364/boe.420953
26
WeigertM.SchmidtU.BootheT.MüllerA.DibrovA.JainA.et al (2018). Content-Aware Image Restoration: pushing the limits of fluorescence microscopy. Nat. Methods15, 1090–1097. 10.1038/s41592-018-0216-7
27
ZhangK.ZuoW.ChenY.MengD.ZhangL. (2017). Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process.26, 3142–3155. 10.1109/tip.2017.2662206
28
ZhangY.GuldnerI. H.NicholsE. L.BenirschkeD.SmithC. J.ZhangS.et al (2021). Instant FLIM enables 4D in vivo lifetime imaging of intact and injured zebrafish and mouse brains. Optica8, 885–897. 10.1364/optica.426870
29
ZhangY.HatoT.DagherP. C.NicholsE. L.SmithC. J.DunnK. W.et al (2019). Automatic segmentation of intravital fluorescence microscopy images by K-means clustering of FLIM phasors. Opt. Lett.44, 3928–3931. 10.1364/ol.44.003928
Summary
Keywords
phasor lifetime synthesis, fluorescence lifetime imaging microscopy (FLIM), lifetime image analysis, convolutional neural networks (CNNs), phasor clustering method, image segmentation, deep learning
Citation
Mannam V, P. Brandt J, Smith CJ, Yuan X and Howard S (2023) Improving fluorescence lifetime imaging microscopy phasor accuracy using convolutional neural networks. Front. Bioinform. 3:1335413. doi: 10.3389/fbinf.2023.1335413
Received
08 November 2023
Accepted
27 November 2023
Published
22 December 2023
Volume
3 - 2023
Edited by
Yide Zhang, California Institute of Technology, United States
Reviewed by
Xin Tong, California Institute of Technology, United States
Binglin Shen, Shenzhen University, China
Updates

Check for updates
Copyright
© 2023 Mannam, P. Brandt, Smith, Yuan and Howard.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Varun Mannam, vmannam@nd.edu; Scott Howard, showard@nd.edu
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.