Image Processing Filters for Grids of Cells Analogous to Filters Processing Grids of Pixels

1 Introduction

Forces between intra-cellular structures, groups of cells and external structures are involved in forming an organism (Stern, 2004; Rauzi et al., 2008; Benton et al., 2013; Münster et al., 2019; Bhide et al., 2020). Processing images using filters for segmenting and analysing sub-cellular structures and cells is a common task for bio-image analysts. Furthermore, the concept of grouping pixels into superpixels for further analysis is well known in computer vision (Ren and Malik, 2003). Thus, it is at hand to process groups of pixels which represent cells and thus, sub-entities of images. Quantitative measurements derived from those digital representations of individual cells can deliver insights about a cells’ identity. However, such measurements may not fully capture important aspects of tissue formation because of their spatially constrained nature. A cells’ identity might depend on properties of neighboring cells (Toth et al., 2018). To unravel physical principles on tissue level, it is worthwhile to formulate image processing filters taking numeric parameters, such as quantitative measurements, of cells and cell-neighbors into account analogously to classical image processing filters taking the intensity of pixels and their neighbors as input.

Most image processing software packages are limited to process imaging data on pixel level using filters. Filters are defined as numerical operations which consume an image to produce another image. Images can be seen as grids of rectangular shaped picture elements, pixels, in 2D or volume elements, voxels, in 3D. Typically, filters take the neighboring pixels of every pixel into account to produce a new pixel value. For example, the mean average filter computes the average intensity of pixels around every pixel and saves the resulting value for each pixel in the new image. In common image processing software, various neighborhood specifications are used: Pixels touching a given reference pixel in corners, edges and voxels touching the given voxel in a plane can be seen as directly touching neighbors. Furthermore, pixels/voxels within a rectangular/cuboidal or ellipsoidal shaped region with given radii around the reference pixel form the proximal neighborhood. Some image processing libraries also offer customizing neighborhood definitions to specific needs. Such a definition is then referred to as structuring element or footprint of a filter.

Combining pixels to objects is common practice and quantification of parameters of objects, such as size and shape, is straightforward. With that, we can visualize parameters of objects as intensity in images, so called parametric images. For example, a parametric image can show large objects with a higher intensity and small objects with a lower intensity. Objects can be cells and thus, studying quantitative parameters of cells in parametric images is straightforward. Studying relationships between cells in such parametric images is more complex and requires some form of so called region adjacency graph (RAG), an abstract representation of neighborhood relationships between cells. For example, measuring the mean area of cells in a given neighborhood around each cell requires determining the RAG of all cells in the tissue and then averaging the area of neighboring cells for each cell. If this would be straightforward, one could think of selecting all cells in a tissue which for example are smaller than the average area minus two times the standard deviation of the area of cells in a given neighborhood. In a more advanced setting, cell classification algorithms could take properties of cells into account that are within a given radius around the cell of interest. Image analysis workflows could benefit from incorporating such operations on tissue level and thus, might enable new perspectives when studying for example tissue development.

Software libraries in R and Python for processing RAGs, such as SPIAT (Yang et al., 2020) and NetworkX (Hagberg et al., 2008), are accessible to an audience with programming experience and offer powerful approaches to study spatial relationships between cells. Widely used open source tools with graphical user interfaces for processing quantitative measurements of cells and their neighbors such as CytoMap (Stoltzfus et al., 2020) typically operate on tabular data and are not integrated in image processing software. Another software package for digital pathology, QuPath (Bankhead et al., 2017), supports similar tools and is limited to two dimensional image data. Thus, the even though operations on RAGs in the cell/tissue context are well known, their application as general purpose tools for processing information between three dimensional cells were yet missing. In this article I present filters for processing objects such as cells, their neighbors and quantitative information derived from them targeting general-purpose use-cases. These filters can process two- and three-dimensional image data and are available for users without programming experience in the ImageJ/Fiji (Schneider et al., 2012; Schindelin et al., 2015) and napari (Sofroniew et al., 2020) ecosystems.

2 Methods

Analogously to image processing filters taking pixel intensities into account, filtering RAGs of cells is feasible. For example, when applying a mean average filter on neighboring cells, the filter could average the intensity, or any other quantitative parameter, of neighboring cells within a certain region and store the resulting value in the corresponding cell of the resulting image. This way you can combine context specific information and any other neighbourhood properties. As for those filters on tissue level the RAG needs to be taken into account, a visual representation of the RAG is required for a user friendly interface. Label images can be used as convenient visual representation of RAGs. Label images are also based on pixel grids and are used to label objects, e.g. with numbers. Thus, in the case of processing images of tissues, a label image could have intensity values numerating the cells. A given pixel intensity 1 means the pixel belongs to cell 1. A label image containing n cells can have pixel intensities ranging from 0 to n, with 0 representing background or empty space between cells. From a label image it is straightforward to retrieve an RAGs following the above specified neighborhood definitions, for example by searching the label image for adjacent pixels with different label values. If there are two neighboring pixels with labels 1 and 2, obviously the cells 1 and 2 are adjacent. In the following, that case will be referred to as “touching neighbors”. As mentioned above, touching neighbors are also defined for pixel grids and thus, the analogy between pixel-based image processing filters and cell-based filters is obvious. For determining if cells are within a given circular neighborhood, the centroids of the cells can be used to compute their distances to each other and to derive an RAG based on cell-cell centroid distances below a given threshold. This case can be referred to as “proximal neighbors”. Two examples for touching and proximal pixels and cells are given in Figure 1. An example workflow applying filters to touching neighbors for the purpose of image segmentation in grids of pixels and grids of cells is shown in Figure 2. Furthermore, for the sake of completeness, when studying relationships between segmented objects such as cells, another type of adjacency is commonly used: n-nearest neighbors. Also in this case, centroids of the cells can be used to determine a given number of cells that are adjacent to every cell in a given label image.

FIGURE 1

FIGURE 1. Neighborhood relationships of pixels (A) and cells (B): Pixels and cells sharing edges can be considered “touching” neighbors. Cells and pixels with a centroid distance below a given threshold can be considered “proximal” neighbors.

FIGURE 2

FIGURE 2. Comparison between filters for binarizing an image on a grid of pixels (A) and a grid of cells (B). Both can be used analogously: When applying a mean average filter on a pixel-grid, the intensities of neighboring pixels are averaged. In case of a cell-grid, intensities of cells in a given parametric image are averaged locally to make a new parametric image. The associated label image serves as representation of the region adjacency graph of the cells. Binary image processing filters such as erosion and dilation work analogously, also on the same region adjacency graph expressing neighborhood relationships between cells.

The proposed implementation offers so called label neighbor filters for processing labeled objects such as cells. The filters include local minimum, mean, maximum and standard deviation. Those filters work analogously to filters operating on a pixel grid. Furthermore, a filter for calculating the mode of neighbors on a cell grid has been implemented. This filter can be used for post-processing of cell classification results. If a classification algorithm for labeled objects results in a classification that is not perfect and rather noisy, the mode filter can reclassify the cells based on the most popular adjacent cell classification. A synthetical use case for this filter is shown in Figure 3.

FIGURE 3

FIGURE 3. Application of the mode filter to a binary parametric image. Starting from a cell grid shown as label image (A) and a corresponding imperfect classification shown as binary parametric image (B), the application of mode filter of proximal cells within a radius of 80 pixels (C) can be used to locally correct classification results. Note that the cells are no hexagons. The simulated tissue consists of cells with variable shape, position and size.

Parametric images, e.g., showing shape, size and topological parameters of cells in tissues, which can be processed using the proposed filters can be generated using the CLIJ (Haase et al., 2020b) and the MorpholibJ (Legland et al., 2016) libraries in the ImageJ/Fiji ecosystem. Those parametric images are can also be used within the python/napari ecosystem. For generating those parametric images with python, programming experience is necessary, e.g., to make use of the scikit-image (van der Walt et al., 2014) library. Also other quantitative image processing libraries can be used. The only constraint is that the parametric images and label images must be available as numpy (Harris et al., 2020) arrays.

To demonstrate the label neighbor filters with a realistic use-case, cell classification in a developing Tribolium castaneum embryo serves as example. Starting point is a three dimensional data set of the embryo and a given cell segmentation. A classifier based on average distances to neighboring cells was trained to differentiate embryo, serosa and a border region in between the two where possibly the amnion will form later during development (Benton et al., 2013). As visualized in Figure 4, the resulting classification is a bit noisy. After filtering the cells for the mode of proximal classes, the cell classification appears more organized spatially. The border region between embryo and serosa can be identified more clearly, which may be crucial for understanding the differentiation of these tissues biologically.

FIGURE 4

FIGURE 4. Application of the mode filter to a classification result of cells in a developing Tribolium castaneum embryo. Starting from a given 3D image dataset (A) and a label image representing the cell segmentation (B), cells can be classified into three classes. If the classification is noisy (C), application of the mode of proximal neighbors filter can be used to determine the most popular class in each cell’s neighborhood (D).

The proposed user interfaces in ImageJ/Fiji and napari offer a category of “Label neighbor filters” containing the above introduced filters as shown in Figure 5 and Figure 6, respectively. The tools are available to end-users as part of the CLIJ2-assistant (Haase et al., 2020a) in Fiji and the napari-pyclesperanto-assistant (Haase et al., 2021) in napari. In Fiji, the tools can be installed by activating the “clij” and “clij2″ update sites. The current development version of the assistant can be installed by activating the “clijx-assistant” update site. The napari-pyclesperanto-assistant can be installed in napari using napari’s the plugin installer. The napari plugin is also under development yet as napari itself is in early developmental stage. Tools that are under development should be used with care and we developers highly value feedback to guide development of our tools in the right direction. Both, the Fiji and the napari plugins, execute the filters based on GPU-accelerated image processing kernels of the CLIJ library. The filters are available in an interactive fashion allowing the users to change parameters, such as the maximum distance of cells to be considered as proximal neighbors, and see the resulting filtered parametric image in real time. Furthermore, in both platforms the user can generate code that represents a workflow as designed on screen and thus, enable reproducibility of the workflow incorporating label neighbor filters potentially together with classical image processing, segmentation, analysis and visualisation steps.

FIGURE 5

FIGURE 5. User interface in Fiji: The label neighborhood filters are available in Fiji via the CLIJ2-assistant user interface on a right-click. After clicking on the given filter, a new window opens and a corresponding dialog where the user can fine tune parameters such as the maximum distance of cells to be considered in the filter as neighborhood.

FIGURE 6

FIGURE 6. User interface in napari: Users can access the label neighborhood filters via the napari-pyclesperanto-assistant user interface. After clicking on a category, the user can select a filter such as mode of proximal neighbors and fine tune parameter such as the maximum distance of cells to be considered for determining the mode.

3 Discussion

The presented label neighbor filters fill a gap in the landscape of available image processing techniques and build a bridge between biologists, bio-image analysts and graph/network scientists. When studying cells and tissues using computational methods, interdisciplinary communication and terminology has to be chosen carefully. Typically, experts studying measurements of cells on tissue level speak about RAGs. Those graphs consist of nodes, which represent cells, and edges which connect nodes and thus, represent the adjacency/neighborhood relationship between cells. Edges could also be seen as abstract representation of tight junctions between two cells. Thus, cell-cell communication, via chemical signals or physical forces, happens along the edge in computational models. While understanding the concepts of graphs, nodes and edges appears crucial from a computer science perspective, a biologist may think more about how cells exchange proteins through their membranes. A physicist may think about forces along cell membranes. Thus, it appears worthwhile to remove any kind of field-specific terminology from a given user interface and present the filters for processing cells and tissues in a generic fashion. As the concepts also work for objects that are no cells, the presented user interfaces use the term “label images” as it is common in the image processing context. Furthermore, while the proposed implementation of the label neighbor filters is limited to basic descriptive statistics operations, the framework is extensible via the ImageJ2 plugin mechanism (Rueden et al., 2017) in Fiji. A solution for extending the napari infrastructure is on the way. Contributions to the underlying open source projects are welcome.

When it comes to storing information such as neighborhood relationships, choosing the right file format is crucial. NetworkX supports a JSON-based file format, which is not compatible to any ImageJ/Fiji or napari plugin at the time of writing this. On Fiji side, plugins for managing lineage trees, which are in practice spatio-temporal RAGs, can be managed using plugins such as Trackmate (Tinevez et al., 2017) and stored as XML-based files. Due to the lack of an accepted community standard for cross-platform file formats for storing RAGs, and no available interactive graphical user interfaces for managing those in the biological image processing context, the proposed software uses label images as interchange format for RAGs. Label images can be stored in manifold file formats such as TIF and manipulated using various software packages including plugins for ImageJ, Fiji and napari. The additional processing step to retrieve an RAG from the label image can be neglected in our implementation as processing is executed on graphic processing units (GPUs). This operation can be parallelized and thus, fits well in the scope of GPU-acceleration libraries such as CLIJ.

4 Conclusion

In this manuscript I presented filters for local filtering of quantitative parameters of segmented and labeled cells within their local environment in two and three dimensional imaging data. Such tools appear crucial for visualization, analysis and deeper understanding of quantitative parameters on tissue level. Special emphasis was put on accessibility and user-friendliness to enable a broad audience to explore their image data of cells and tissues.

Data Availability Statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: https://github.com/haesleinhuepf/label_neighbor_filters.

Author Contributions

RH developed the software, made the figures and wrote the manuscript.

Funding

I acknowledge support by the Deutsche Forschungsgemeinschaft under Germany’s Excellence Strategy—EXC2068–Cluster of Excellence Physics of Life of TU Dresden.

Conflict of Interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

I would like to thank Daniela Vorkel, MPI-CBG/CSBD Dresden, for imaging the used Tribolium embryo data and Nicola Maghelli and Gene Myers, MPI-CBG/CSBD Dresden, for providing the imaging infrastucture. Furthermore, I thank Christopher Schmied, Human Technopole Milan, and Pradeep Rajasekhar, WEHI Melbourne, for reading the manuscript and providing helpful feedback. Most importantly, I thank Walter Pätzold, HTW Dresden, because he gave an inspring lecture about the topic 15 years ago.

References

Bankhead, P., Loughrey, M. B., Fernández, J. A., Dombrowski, Y., McArt, D. G., Dunne, P. D., et al. (2017). Qupath: Open Source Software for Digital Pathology Image Analysis. Sci. Rep. 7, 16878. doi:10.1038/s41598-017-17204-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Benton, M. A., Akam, M., and Pavlopoulos, A. (2013). Cell and Tissue Dynamics during tribolium Embryogenesis Revealed by Versatile Fluorescence Labeling Approaches. Development 140, 3210–3220. doi:10.1242/dev.096271

PubMed Abstract | CrossRef Full Text | Google Scholar

Bhide, S., Gombalova, D., Mönke, G., Stegmaier, J., Zinchenko, V., Kreshuk, A., et al. (2020). Mechanical Competition Alters the Cellular Interpretation of an Endogenous Genetic Programme. bioRxiv. 220. e202104107. doi:10.1101/2020.10.15.333963

CrossRef Full Text | Google Scholar

Haase, R., Jain, A., Rigaud, S., Vorkel, D., Rajasekhar, P., Suckert, T., et al. (2020a). Interactive Design of Gpu-Accelerated Image Data Flow Graphs and Cross-Platform Deployment Using Multi-Lingual Code Generation. bioRxiv. doi:10.1101/2020.11.19.386565

CrossRef Full Text | Google Scholar

Haase, R., Lambert, T., and Kiggins, J. (2021). Napari-Pyclesperanto-Assistant, 1. doi:10.5281/zenodo.5501317

CrossRef Full Text | Google Scholar

Haase, R., Royer, L. A., Steinbach, P., Schmidt, D., Dibrov, A., Schmidt, U., et al. (2020b). Clij: Gpu-Accelerated Image Processing for Everyone. Nat. Methods 17, 5–6. doi:10.1038/s41592-019-0650-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Hagberg, A. A., Schult, D. A., and Swart, P. J. (2008). “Exploring Network Structure, Dynamics, and Function Using Networkx,” in Proceedings of the 7th Python in Science Conference. Editors G. Varoquaux, T. Vaught, and J. Millman (, CA USA), 11–15.

Google Scholar

Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., et al. (2020). Array Programming with NumPy. Nature 585, 357–362. doi:10.1038/s41586-020-2649-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Legland, D., Arganda-Carreras, I., and Andrey, P. (2016). MorphoLibJ: Integrated Library and Plugins for Mathematical Morphology with ImageJ. Bioinformatics 32, btw413–3534. doi:10.1093/bioinformatics/btw413

PubMed Abstract | CrossRef Full Text | Google Scholar

Münster, S., Jain, A., Mietke, A., Pavlopoulos, A., Grill, S. W., and Tomancak, P. (2019). Attachment of the Blastoderm to the Vitelline Envelope Affects Gastrulation of Insects. Nature 568, 395–399. doi:10.1038/s41586-019-1044-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Rauzi, M., Verant, P., Lecuit, T., and Lenne, P.-F. (2008). Nature and Anisotropy of Cortical Forces Orienting drosophila Tissue Morphogenesis. Nat. Cel Biol. 10, 1401–1410. doi:10.1038/ncb1798

PubMed Abstract | CrossRef Full Text | Google Scholar

Ren, X., and Malik, J. (2003). Learning a Classification Model for Segmentation. Proc. Ninth IEEE Int. Conf. Comp. Vis. 1, 10–17. doi:10.1109/ICCV.2003.1238308

CrossRef Full Text | Google Scholar

Rueden, C. T., Schindelin, J., Hiner, M. C., DeZonia, B. E., Walter, A. E., Arena, E. T., et al. (2017). Imagej2: Imagej for the Next Generation of Scientific Image Data. BMC Bioinformatics 18, 529. doi:10.1186/s12859-017-1934-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Schindelin, J., Rueden, C. T., Hiner, M. C., and Eliceiri, K. W. (2015). The Imagej Ecosystem: An Open Platform for Biomedical Image Analysis. Mol. Reprod. Dev. 82, 518–529. doi:10.1002/mrd.22489

PubMed Abstract | CrossRef Full Text | Google Scholar

Schneider, C. A., Rasband, W. S., and Eliceiri, K. W. (2012). Nih Image to Imagej: 25 Years of Image Analysis. Nat. Methods 9, 671–675. doi:10.1038/nmeth.2089

PubMed Abstract | CrossRef Full Text | Google Scholar

Sofroniew, N., Lambert, T., Evans, K., Nunez-Iglesias, J., Yamauchi, K., Solak, A. C., et al. (2020). Napari: Available at: https://zenodo.org/record/4048613. accessed 2020-10-31. doi:10.5281/ZENODO.4048613[Dataset]

CrossRef Full Text | Google Scholar

Stern, C. D. (2004). Gastrulation: From Cells to Embryo. Cold Spring Harbor Laboratory Press.

Google Scholar

Stoltzfus, C. R., Filipek, J., Gern, B. H., Olin, B. E., Leal, J. M., Wu, Y., et al. (2020). Cytomap: A Spatial Analysis Toolbox Reveals Features of Myeloid Cell Organization in Lymphoid Tissues. Cel Rep. 31, 107523. doi:10.1016/j.celrep.2020.107523

PubMed Abstract | CrossRef Full Text | Google Scholar

Tinevez, J.-Y., Perry, N., Schindelin, J., Hoopes, G. M., Reynolds, G. D., Laplantine, E., et al. (2017). Trackmate: An Open and Extensible Platform for Single-Particle Tracking. Methods 115, 80–90. doi:10.1016/j.ymeth.2016.09.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Toth, T., Balassa, T., Bara, N., Kovacs, F., Kriston, A., Molnar, C., et al. (2018). Environmental Properties of Cells Improve Machine Learning-Based Phenotype Recognition Accuracy. Sci. Rep. 8, 10085. doi:10.1038/s41598-018-28482-y

PubMed Abstract | CrossRef Full Text | Google Scholar

van der Walt, S., Schönberger, J. L., Nunez-Iglesias, J., Boulogne, F., Warner, J. D., Yager, N., et al. (2014). Scikit-Image: Image Processing in Python. PeerJ 2, e453. doi:10.7717/peerj.453

PubMed Abstract | CrossRef Full Text | Google Scholar

Yang, T., Ozcoban, V., Pasam, A., Kocovski, N., Pizzolla, A., Huang, Y.-K., et al. (2020). Spiat: An R Package for the Spatial Image Analysis of Cells in Tissues. bioRxiv. doi:10.1101/2020.05.28.122614

CrossRef Full Text | Google Scholar