Image Processing Filters for Grids of Cells Analogous to Filters Processing Grids of Pixels

Intra- and extra-cellular processes shape tissues together. For understanding how neighborhood relationships between cells play a role in this process, having image processing filters based on these relationships would be beneficial. Those operations are known and their application to microscopy image data typically requires programming skills. User-friendly general purpose tools for pursuing image processing on a level of neighboring cells were yet missing. In this manuscript I demonstrate image processing filters which process grids of cells on tissue level and the analogy to their better known counter parts processing grids of pixels. The tools are available as part of free and open source software in the ImageJ/Fiji and napari ecosystems and their application does not require any programming experience.


INTRODUCTION
Forces between intra-cellular structures, groups of cells and external structures are involved in forming an organism (Stern, 2004;Rauzi et al., 2008;Benton et al., 2013;Münster et al., 2019;Bhide et al., 2020). Processing images using filters for segmenting and analysing sub-cellular structures and cells is a common task for bio-image analysts. Furthermore, the concept of grouping pixels into superpixels for further analysis is well known in computer vision (Ren and Malik, 2003). Thus, it is at hand to process groups of pixels which represent cells and thus, sub-entities of images. Quantitative measurements derived from those digital representations of individual cells can deliver insights about a cells' identity. However, such measurements may not fully capture important aspects of tissue formation because of their spatially constrained nature. A cells' identity might depend on properties of neighboring cells (Toth et al., 2018). To unravel physical principles on tissue level, it is worthwhile to formulate image processing filters taking numeric parameters, such as quantitative measurements, of cells and cell-neighbors into account analogously to classical image processing filters taking the intensity of pixels and their neighbors as input.
Most image processing software packages are limited to process imaging data on pixel level using filters. Filters are defined as numerical operations which consume an image to produce another image. Images can be seen as grids of rectangular shaped picture elements, pixels, in 2D or volume elements, voxels, in 3D. Typically, filters take the neighboring pixels of every pixel into account to produce a new pixel value. For example, the mean average filter computes the average intensity of pixels around every pixel and saves the resulting value for each pixel in the new image. In common image processing software, various neighborhood specifications are used: Pixels touching a given reference pixel in corners, edges and voxels touching the given voxel in a plane can be seen as directly touching neighbors. Furthermore, pixels/voxels within a rectangular/cuboidal or ellipsoidal shaped region with given radii around the reference pixel form the proximal neighborhood. Some image processing libraries also offer customizing neighborhood definitions to specific needs. Such a definition is then referred to as structuring element or footprint of a filter.
Combining pixels to objects is common practice and quantification of parameters of objects, such as size and shape, is straightforward. With that, we can visualize parameters of objects as intensity in images, so called parametric images. For example, a parametric image can show large objects with a higher intensity and small objects with a lower intensity. Objects can be cells and thus, studying quantitative parameters of cells in parametric images is straightforward. Studying relationships between cells in such parametric images is more complex and requires some form of so called region adjacency graph (RAG), an abstract representation of neighborhood relationships between cells. For example, measuring the mean area of cells in a given neighborhood around each cell requires determining the RAG of all cells in the tissue and then averaging the area of neighboring cells for each cell. If this would be straightforward, one could think of selecting all cells in a tissue which for example are smaller than the average area minus two times the standard deviation of the area of cells in a given neighborhood. In a more advanced setting, cell classification algorithms could take properties of cells into account that are within a given radius around the cell of interest. Image analysis workflows could benefit from incorporating such operations on tissue level and thus, might enable new perspectives when studying for example tissue development.
Software libraries in R and Python for processing RAGs, such as SPIAT (Yang et al., 2020) and NetworkX (Hagberg et al., 2008), are accessible to an audience with programming experience and offer powerful approaches to study spatial relationships between cells. Widely used open source tools with graphical user interfaces for processing quantitative measurements of cells and their neighbors such as CytoMap (Stoltzfus et al., 2020) typically operate on tabular data and are not integrated in image processing software. Another software package for digital pathology, QuPath (Bankhead et al., 2017), supports similar tools and is limited to two dimensional image data. Thus, the even though operations on RAGs in the cell/ tissue context are well known, their application as general purpose tools for processing information between three dimensional cells were yet missing. In this article I present filters for processing objects such as cells, their neighbors and quantitative information derived from them targeting generalpurpose use-cases. These filters can process two-and threedimensional image data and are available for users without programming experience in the ImageJ/Fiji (Schneider et al., 2012;Schindelin et al., 2015) and napari (Sofroniew et al., 2020) ecosystems.
FIGURE 1 | Neighborhood relationships of pixels (A) and cells (B): Pixels and cells sharing edges can be considered "touching" neighbors. Cells and pixels with a centroid distance below a given threshold can be considered "proximal" neighbors.
Frontiers in Computer Science | www.frontiersin.org November 2021 | Volume 3 | Article 774396 FIGURE 2 | Comparison between filters for binarizing an image on a grid of pixels (A) and a grid of cells (B). Both can be used analogously: When applying a mean average filter on a pixel-grid, the intensities of neighboring pixels are averaged. In case of a cell-grid, intensities of cells in a given parametric image are averaged locally to make a new parametric image. The associated label image serves as representation of the region adjacency graph of the cells. Binary image processing filters such as erosion and dilation work analogously, also on the same region adjacency graph expressing neighborhood relationships between cells.

METHODS
Analogously to image processing filters taking pixel intensities into account, filtering RAGs of cells is feasible. For example, when applying a mean average filter on neighboring cells, the filter could average the intensity, or any other quantitative parameter, of neighboring cells within a certain region and store the resulting value in the corresponding cell of the resulting image. This way you can combine context specific information and any other neighbourhood properties. As for those filters on tissue level the RAG needs to be taken into account, a visual representation of the RAG is required for a user friendly interface. Label images can be used as convenient visual representation of RAGs. Label images are also based on pixel grids and are used to label objects, e.g. with numbers. Thus, in the case of processing images of tissues, a label image could have intensity values numerating the cells. A given pixel intensity 1 means the pixel belongs to cell 1. A label image containing n cells can have pixel intensities ranging from 0 to n, with 0 representing background or empty space between cells. From a label image it is straightforward to retrieve an RAGs following the above specified neighborhood definitions, for example by searching the label image for adjacent pixels with different label values. If there are two neighboring pixels with labels 1 and 2, obviously the cells 1 and 2 are adjacent. In the following, that case will be referred to as "touching neighbors". As mentioned above, touching neighbors are also defined for pixel grids and thus, the analogy between pixel-based image processing filters and cell-based filters is obvious. For determining if cells are within a given circular neighborhood, the centroids of the cells can be used to compute their distances to each other and to derive an RAG based on cell-cell centroid distances below a given threshold. This case can be referred to as "proximal neighbors". Two examples for touching and proximal pixels and cells are given in Figure 1. An example workflow applying filters to touching neighbors for the purpose of image segmentation in grids of pixels and grids of cells is shown in Figure 2. Furthermore, for the sake of completeness, when studying relationships between segmented objects such as cells, another type of adjacency is commonly used: n-nearest neighbors. Also in this case, centroids of the cells can be used to determine a given number of cells that are adjacent to every cell in a given label image.
The proposed implementation offers so called label neighbor filters for processing labeled objects such as cells. The filters include local minimum, mean, maximum and standard deviation. Those filters work analogously to filters operating on a pixel grid. Furthermore, a filter for calculating the mode of neighbors on a cell grid has been implemented. This filter can be used for postprocessing of cell classification results. If a classification algorithm for labeled objects results in a classification that is not perfect and rather noisy, the mode filter can reclassify the cells based on the most popular adjacent cell classification. A synthetical use case for this filter is shown in Figure 3.
Parametric images, e.g., showing shape, size and topological parameters of cells in tissues, which can be processed using the proposed filters can be generated using the CLIJ (Haase et al., 2020b) and the MorpholibJ (Legland et al., 2016) libraries in the ImageJ/Fiji ecosystem. Those parametric images are can also be used within the python/napari ecosystem. For generating those parametric images with python, programming experience is necessary, e.g., to make use of the scikit-image (van der Walt et al., 2014) library. Also other quantitative image processing libraries can be used. The only constraint is that the parametric images and label images must be available as numpy (Harris et al., 2020) arrays.
To demonstrate the label neighbor filters with a realistic usecase, cell classification in a developing Tribolium castaneum embryo serves as example. Starting point is a three dimensional data set of the embryo and a given cell segmentation. A classifier based on average distances to neighboring cells was trained to differentiate embryo, serosa and a border region in between the two where possibly the amnion will form later during development (Benton et al., 2013). As visualized in Figure 4, the resulting classification is a bit noisy. After filtering the cells for the mode of proximal classes, the cell classification appears more organized spatially. The border region between embryo and serosa can be identified more clearly, which may be crucial for understanding the differentiation of these tissues biologically.
The proposed user interfaces in ImageJ/Fiji and napari offer a category of "Label neighbor filters" containing the above introduced filters as shown in Figure 5 and Figure 6, respectively. The tools are available to end-users as part of the CLIJ2-assistant (Haase et al., 2020a) in Fiji and the napari-pyclesperanto-assistant (Haase et al., 2021) in napari. In Fiji, the tools can be installed by activating the "clij" and "clij2″ update sites. The current development version of the assistant can be installed by activating the "clijx-assistant" update site. The napari-pyclesperanto-assistant can be installed in napari using napari's the plugin installer. The napari plugin is also under development yet as napari itself is in early developmental stage. Tools that are under development should be used with care and we developers highly value feedback to guide development of our tools in the right direction. Both, the Fiji and the napari plugins, execute the filters based on GPU-accelerated image processing kernels of the CLIJ library. The filters are available in an interactive fashion allowing the users to change parameters, such as the maximum distance of cells to be considered as proximal neighbors, and see the resulting filtered parametric image in real time. Furthermore, in both platforms the user can generate code that represents a workflow as designed on screen and thus, enable reproducibility of the workflow incorporating label neighbor filters potentially together with classical image processing, segmentation, analysis and visualisation steps. Frontiers in Computer Science | www.frontiersin.org November 2021 | Volume 3 | Article 774396

DISCUSSION
The presented label neighbor filters fill a gap in the landscape of available image processing techniques and build a bridge between biologists, bio-image analysts and graph/network scientists. When studying cells and tissues using computational methods, interdisciplinary communication and terminology has to be chosen carefully. Typically, experts studying measurements of cells on tissue level speak about RAGs. Those graphs consist of nodes, which represent cells, and edges which connect nodes and thus, represent the adjacency/neighborhood relationship between cells. Edges could also be seen as abstract representation of tight junctions between two cells. Thus, cell-cell communication, via chemical signals or physical forces, happens along the edge in computational models. While understanding the concepts of graphs, nodes and edges appears crucial from a computer science perspective, a biologist may think more about how cells exchange proteins through their membranes. A physicist may think about forces along cell membranes. Thus, it appears worthwhile to remove any kind of field-specific terminology from a given user interface and present the filters for processing cells and tissues in a generic fashion. As the concepts also work for objects that are no cells, the presented user interfaces use the term "label images" as it is common in the image processing context. Furthermore, while the proposed implementation of the label neighbor filters is limited to basic descriptive statistics operations, the framework is extensible via the ImageJ2 plugin mechanism (Rueden et al., 2017) in Fiji. A solution for extending the napari infrastructure is on the way. Contributions to the underlying open source projects are welcome.
When it comes to storing information such as neighborhood relationships, choosing the right file format is crucial. NetworkX supports a JSON-based file format, which is not compatible to any ImageJ/Fiji or napari plugin at the time of writing this. On Fiji side, plugins for managing lineage trees, which are in practice spatiotemporal RAGs, can be managed using plugins such as Trackmate (Tinevez et al., 2017) and stored as XML-based files. Due to the lack of an accepted community standard for cross-platform file formats for storing RAGs, and no available interactive graphical user interfaces for managing those in the biological image processing context, the proposed software uses label images as interchange format for RAGs. Label images can be stored in manifold file formats such as TIF and manipulated using various software packages including plugins for ImageJ, Fiji and napari. The additional processing step to retrieve an RAG from the label image can be neglected in our implementation as processing is executed on graphic processing units (GPUs). This operation can be parallelized and thus, fits well in the scope of GPU-acceleration libraries such as CLIJ.

CONCLUSION
In this manuscript I presented filters for local filtering of quantitative parameters of segmented and labeled cells within their local FIGURE 6 | User interface in napari: Users can access the label neighborhood filters via the napari-pyclesperanto-assistant user interface. After clicking on a category, the user can select a filter such as mode of proximal neighbors and fine tune parameter such as the maximum distance of cells to be considered for determining the mode.
Frontiers in Computer Science | www.frontiersin.org November 2021 | Volume 3 | Article 774396 environment in two and three dimensional imaging data. Such tools appear crucial for visualization, analysis and deeper understanding of quantitative parameters on tissue level. Special emphasis was put on accessibility and user-friendliness to enable a broad audience to explore their image data of cells and tissues.

DATA AVAILABILITY STATEMENT
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: https://github.com/ haesleinhuepf/label_neighbor_filters.