Leveraging a Foundation Model Zoo for Cell Similarity Search in Oncological Microscopy across Devices

Gabriel Kalweit^1,2*Anusha Klett¹Paula Silvestrini^1,3Jens Rahnfeld^1,2Mehdi Naouar^1,2Yannick Vogt^1,2Diana Infante^1,4Rebecca Berger⁴Jesús Duque-Afonso⁴Tanja Nicole Hartmann⁴Marie Follo^4,5Elitsa Bodurova-Spassova^4,5Michael Lübbert^4,6Roland Mertelsmann^1,4,7Joschka Boedecker^1,2,8Evelyn Ullrich^{10,11,12,13,9}Maria Kalweit^1,2

• ¹Collaborative Research Institute Intelligent Oncology, Freiburg, Germany
• ²Neurorobotics Lab, Department of Computer Science, University of Freiburg, Freiburg, Germany
• ³Laboratory of Applied Cellular and Molecular Biology,Institute of Veterinary Sciences of the Litoral,Universidad Nacional del Litoral (UNL)-National Scientific and Technical Research Council (CONICET), Esperanza, Argentina
• ⁴Department of Hematology, Oncology and Stem Cell Transplantation, Medical Center University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
• ⁵Lighthouse Core Facility, Medical Center – University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
• ⁶German Cancer Consortium (DKTK) and German Cancer Research Center (DKFZ), Partner Site Freiburg, Freiburg, Germany
• ⁷Mertelsmann Foundation, Freiburg, Germany
• ⁸IMBIT//BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
• ⁹Department of Pediatrics, Experimental Immnology and Cell Therapy, Goethe University Frankfurt, Frankfurt am Main, Germany
• ¹⁰University Cancer Center Frankfurt (UCT), Frankfurt am Main, Germany
• ¹¹Mildred Scheel Career Center (MSNZ), Hospital of the Goethe University, Frankfurt, Hesse, Germany
• ¹²German Cancer Consortium (DKTK) and German Cancer Research Center (DKFZ), Partner Site Frankfurt, Frankfurt am Main, Germany
• ¹³Frankfurt Cancer Institute (FCI), Frankfurt, Hesse, Germany

Background: Cellular imaging analysis using the traditional retrospective approach is extremely time-consuming and labor-intensive. Although AI-based solutions are available, these approaches rely heavily on supervised learning techniques that require high quality, large labeled datasets from the same microscope to be reliable. In addition, primary patient samples are often heterogeneous cell populations and need to be stained to distinguish the cellular subsets. The resulting imaging data is analyzed and labeled manually by experts. Therefore, a method to distinguish cell populations across imaging devices without the need for staining and extensive manual labeling would help immensely to gain real-time insights into cell population dynamics. This especially holds true for recognizing specific cell types and states in response to treatments. We aim to develop an unsupervised approach using general vision foundation models trained on diverse and extensive imaging datasets to extract rich visual features for cell-analysis across devices, including both stained and unstained live cells. Our method, Entropy-guided Weighted Combinational FAISS (EWC-FAISS), uses these models purely in an inference-only mode without task-specific retraining on the cellular data. Combining the generated embeddings in an efficient and adaptive k-nearest neighbor search allows for automated, crossdevice identification of cell types and states, providing a strong basis for AI-assisted cancer therapy. We utilized two publicly available datasets. The WBC dataset includes 14,424 images of stained white blood cell samples from patients with acute myeloid and lymphoid leukemia, as well as those without leukemic pathology. The LISC dataset comprises 257 images of white blood cell samples from healthy individuals. We generated four in-house datasets utilizing the JIMT-1 breast cancer cell line, as well as Jurkat and K562 (leukemic cell lines). These datasets were acquired using the Nanolive 3D Cell Explorer-fluo (CX-A) holotomographic microscope and the BioTek Lionheart FX automated brightfield microscope. To generate the embeddings, we used and optimized a concatenated combination of SAM, DINO, ConvNeXT, SWIN, CLIP and ViTMAE. The combined embeddings were used as input for the adaptive k-nearest neighbor search, building an approximate Hierarchical Navigable Small World FAISS index. We compared EWC-FAISS to fully trained Classifiers with DINO, SWIN and ConvNeXT backbones, and to NMTune.