Self-supervised learning analysis of multi-FISH labeled cell-type map in thick brain slices

Weijie Zheng^1,2Yiping An^2,3Kang Li³Jinyue Wang²Jianqing Gao^4*Huawei Mu^2,3*Jin Tang^1,2*Hao Wang^2,3*

• ¹Anhui University, Hefei, China
• ²Hefei Comprehensive National Science Center, Hefei, China
• ³University of Science and Technology of China, Hefei, China
• ⁴iFlytek Research, Hefei, China

Accurate mapping of the spatial distribution of diverse cell types is essential for understanding the cellular organization of brain. However, the cellular heterogeneity and the substantial cost of manual annotation of cells in volumetric images hinder existing neural networks from achieving high-precision segmentation of multiple cell-types within a unified framework. To address this challenge, we introduce a self-supervised learning framework Voxel-wise U-shaped Swin-Mamba network (VUSMamba) for automatic segmentation of multiple neuronal populations in 300 µm thick brain slices. As a proof of concept, we applied the framework to a multi-cell-type dataset obtained using multiplexed fluorescence in situ hybridization (multi-FISH) combined with highspeed volumetric microscopy VISoR. VUSMamba employs contrastive learning and pretext tasks for self-supervised learning on unlabeled data, followed by fine-tuning with minimal annotations.Compared to state-of-the-art baseline models, VUSMamba achieves higher segmentation accuracy with reduced computational cost. This work presents a unified self-supervised neural network framework that enables simultaneous high-precision segmentation of glutamatergic neurons, GABAergic neurons, and nuclei, offering a standardized pipeline for constructing and analyzing whole-brain cell-type atlases.