This article was submitted to Computer Vision, a section of the journal Frontiers in Computer Science
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Image analysis workflows for Histology increasingly require the correlation and combination of measurements across several whole slide images. Indeed, for multiplexing, as well as multimodal imaging, it is indispensable that the same sample is imaged multiple times, either through various systems for multimodal imaging, or using the same system but throughout rounds of sample manipulation (e.g. multiple staining sessions). In both cases slight deformations from one image to another are unavoidable, leading to an imperfect superimposition Redundant and thus a loss of accuracy making it difficult to link measurements, in particular at the cellular level. Using pre-existing software components and developing missing ones, we propose a user-friendly workflow which facilitates the nonlinear registration of whole slide images in order to reach sub-cellular resolution level. The set of whole slide images to register and analyze is at first defined as a QuPath project. Fiji is then used to open the QuPath project and perform the registrations. Each registration is automated by using an elastix backend, or semi-automated by using BigWarp in order to interactively correct the results of the automated registration. These transformations can then be retrieved in QuPath to transfer any regions of interest from an image to the corresponding registered images. In addition, the transformations can be applied in QuPath to produce on-the-fly transformed images that can be displayed on top of the reference image. Thus, relevant data can be combined and analyzed throughout all registered slides, facilitating the analysis of correlative results for multiplexed and multimodal imaging.
Whole slide imaging (WSI) is the process of acquiring images of tissue sections. It is typically realized in a line or a tile by tile scanning fashion in order to allow the imaging and visualization of samples exceeding the field of view of a conventional light microscope setup. Most of the commercially available systems accommodate at least two modalities: brightfield and fluorescence imaging. Within the last decade WSI turned out to become more and more important in pathology (
Image processing of WSI images is challenging in many ways, in particular when aiming to register images from different modalities. Images often consist of gigapixel 2D planes, and matching correctly one slide to another usually requires a deformation more complex than a single affine transformation. Besides pure technical problems, life scientists face practical challenges if they want to use published methods. For instance, a grand challenge of non-linear image registration has been performed in 2018 (
Based on our group’s experience as a microscopy core facility imaging, the following combination of requirements is until now not met for non-linear WSI registration: open-source, easy to install, capable to read and display multiresolution image files without conversion, capable to freely deform and interactively display deformed images, giving reasonably fast and accurate results, and compatible with a modular and powerful analysis platform.
The individual tasks and requirements mentioned above are however met in different pieces of software: • For multiresolution file format reading, the OME consortium has built a library, Bio-Formats ( • QuPath ( • Thanks to the ImgLib2 ( • For automated multi-modal registration, many open source libraries exist. We decided to use the powerful and well-established elastix library (
In the following we present a fast and easy to use workflow for whole slide image registration. QuPath, a widely used open-source project for whole slide image display and analysis, serves as the entry point and can be used for further downstream analysis after the registration. This workflow is targeted towards life scientists lowering the entry barrier for whole slide registration and making it more user-friendly and reproducible.
As mentioned above, all the core open source elements needed to build a non-linear WSI registration workflow already exist, however, they do suffer from a lack of interoperability. For instance elastix is a C++ library with python wrappers, without pre-compiled Java wrappers. Moreover, while Fiji and QuPath are both Java-based software, they are incompatible in terms of Java versions. It is difficult however to find and allocate resources in order to make the connections between these components. Nevertheless, bridging these components is needed for life scientists, and this work is the result of such an effort. To bring these components together and provide a complete WSI non-linear registration workflow, the following missing components have been identified and implemented: • A Fiji/BigDataViewer opener for QuPath projects • Support of Bio-Formats multi-resolution API for BigDataViewer • Fiji plugins to bridge elastix and Fiji • A way to apply, store and reopen the deformation function which makes the correspondence between coordinates of registered images.
This work provides these missing pieces and thus closes the gap to a complete open-source workflow for non-linear WSI registration. It starts from the opening/display/definition of the images down to providing modular tools for the analysis in QuPath. An overview of the workflow is presented in
Workflow overview. The set of images to be registered are all put into a single QuPath project. Registrations are performed in Fiji, images are opened from a QuPath project and each registration result is stored as a file within the Redundant project entry folder. For the analysis, thanks to the registration result found between two images, regions of interest can be transferred in QuPath from one image to another, in order to generate correlated data. It is additionally possible to create a new combined image within QuPath. Software components implemented for the workflow are labelled in red. 1: opening QuPath project in BigDataViewer; 2: registering workflow in Fiji, 3: bridging Fiji and elastix; 4: Exporting of transformation file; 5: transferring object from one image to another; 6: generation of combined registered images in QuPath.
In short, all images which need to be registered need to be put into a single QuPath project, which is then opened in BigDataViewer. The reference image is used to align and transform all the other images. The non-reference images will be referred to as the moving images. Pairs of reference and moving images are then registered within Fiji, and transformations are exported to QuPath. The analysis and correlation between images can be performed in QuPath by transforming regions either towards the reference or the moving image. It is also possible to generate an image combining the reference and registered images into a new QuPath entry.
The registration workflow, which happens within Fiji is described in
Registration workflow steps and registration accuracy.
is a manual rigid registration step which is required if the images are not approximately aligned initially. This may happen because Bio-Formats uses the stage location of the microscope to position the images in physical space. If the images to register originate from the same microscope, keeping this offset may be helpful. In other cases, there’s a chance that images won’t overlap at all. The Bio-Formats induced offset can thus be removed at the start of the workflow, and the user can furthermore rigidly move the moving image to improve the alignment before resuming the next steps.
consists of an automated affine registration with elastix (see elastix parameters in
consists of a semi automated spline registration. For this step, a set of user-defined patches are each registered automatically in an affine manner (using the same elastix parameters as step 2). With the default parameters, each patch is a 500 × 500 µm area resampled at 1 µm per pixel. This xy sampling is typically sufficient for images from histological slides where objectives often have a numerical aperture below 1.0. For an accurate registration, these patches should be content-rich, and approximately correctly aligned, a condition met if the first coarse registration is successful. To extend the patches registration over the whole image, only the location of the central point (moving/fixed) of each patch is kept to build a landmark based registration. These points are used to extrapolate the transformation over the whole slide using B-Splines of the ImgLib2 library, as in the BigWarp plugin (
consists of manual spline correction. This step calls the BigWarp plugin ( At the end of the workflow, a transformation file defining how coordinates can be converted from the moving image to the reference image is stored within the QuPath reference image entry folder. In order to demonstrate the performance of this workflow, we applied these four consecutive steps on a test dataset. It consists of a section of a fixed mouse duodenum. Dividing cells have been stained consecutively with two different methods. The fluorescence based 5-Ethynyl-2′-deoxyuridine (EdU) staining with click chemistry was followed by a 3,3′-Diaminobenzidine (DAB) revealing using Anti-Bromodeoxyuridine (BrdU) antibodies, leading to the immunohistochemistry (IHC) image. Each staining step required manipulation of the sample (see materials and methods). This sample requires alignment and registration because it consists of two multi modal images (one fluorescent, one IHC) where different zones of the sample were imaged, with different cameras, and some stitching artefacts are present. However, note that the exact same cells are imaged in the different modalities, which is a requirement for this workflow (i.e. it has not been tested for serial sections). Since the same (dividing) cells are targeted in both images, we expect a very good match between the positively detected cells in both modalities. To assess the quality of the registration in the different steps of the workflow we compare against a ground-truth registration using BigWarp by identifying and placing more than 3,000 landmarks manually, covering the whole slide on identifiable features, in most of the cases individual cells. Making the ground truth required around 6 h of work. Note that on average the uncertainties on the placement of landmarks for the ground truth is of about 1 µm, leading to some uncertainties even in the deformation ground truth. We then assessed the quality of the registration at each step of the workflow, by computing the distance of these landmarks to their target location in the ground truth ( In step 1, we can see that a 10 µm shift is present and this is clearly not precise enough for individual cells to match. Conversely, as in the ground truth, it is hard to distinguish meaningful differences when landmarks are less than 2 µm from their ground truth location. Based on
For downstream analysis in QuPath, we decided to provide a way to transform a region of interest from one slide to another. Since regions of interest are defined with double precision coordinates, very little precision is lost when transforming coordinates. Conversely, generating a transformed image in QuPath creates some interpolation artefacts due to the pixel-based resampling. However, it may be useful to display transformed images over the reference image, e.g. for display and presentation purposes. This can be done with an additional QuPath extension provided in this workflow, called Image Combiner Warpy. With this tool, the transformed images can be overlaid on top of the reference image, based on the transform files and by using various possible sampling methods. Since this overlay process creates only one additional QuPath project entry, using the existing image data, it is executed almost immediately, with no delay time. If effects due to sampling artifacts do not matter, the combined image can be used for analysis in QuPath, including the transformed channels as well as the reference channel. If computation time and storage space are not an issue, and availability as a whole slide image file is important, the combined image project can be fully computed and resaved from within QuPath, for example as an OME tiff. In We assess the distance of a registration to the ground truth by counting the number of cells which differ in this count ( In summary the comparison of the DAB with the fluorescence images demonstrate that this workflow is able to reach high quality registration. Only 1.5% (12 out of 790) of the double positive cells were missed due to alignment errors.
Comparison of cell classes depending on registration procedure. Counting of cell DAB/EdU classes (+/+, −/−, +/−, −/+) for the ground truth and for the different registration methods used (see
DAB+ / EdU+ | DAB- / EdU- | DAB+/EdU- | DAB-/EdU+ | ||
Ground Truth | 790 | 3,353 | 160 | 90 | |
1 | Manual Rigid | 386 (−404) | 3,020 (-333) | 493 (+333) | 494 (+404) |
2 | (1) + Auto Affine | 757 (−33) | 3,279 (-74) | 234 (+74) | 123 (+33) |
3 | (2) + Auto Spline | 767 (−23) | 3,332 (-21) | 181 (+21) | 113 (+23) |
4 | (3) + Manual Spline | 778 (−12) | 3,341 (-12) | 172 (+12) | 102 (+12) |
This workflow has been tested for the applications achieving multiplexing by imaging the same sample through several rounds of staining. As the physical size of the image and the target resolution are known, reasonable default parameters for elastix can be used providing a user interface which is almost parameter free. This is particularly important as parameter fine tuning is cumbersome and challenging for non-experts. Omitting this step clearly lowers the entry barrier for the broader life-science community.
Until now this workflow has not been tested for the registration of serial sections where cellular matching from one slice to the next cannot be expected. Conversely in its present form this workflow is not of sufficient precision to match small sub cellular structures, such as membranous organelles. However, due to its modular design, this workflow can be adapted either by using different alignment parameters or by using other alignment methods, opening the way to support other use cases. In particular, elastix is the library we chose for registrations, but other libraries exist, including ones which are only available in python. By using PyImageJ (
We implemented a workflow for semi-automated non-linear WSI registration which requires minimal user input and can be realized in a few minutes per slide. It facilitates the registration of images for all applications where multiplexing is achieved via several rounds of staining. The performance is almost identical to that of a registration workflow based on manual annotation, but requires far fewer landmarks with the advantages of omitting the time consuming and tedious task of finding landmarks. Therefore it can save several hours of human annotation workload. Thanks to its modular design, it will be extended in the future to support other registration methods and other use cases, such as serial sections alignment (between them or with an atlas) and CLEM.
Sample used for the demo registration (
All the tutorials (written and video) necessary to apply the workflow in practice are accessible in
The dataset presented in this study can be found in an online repository:
The studies involving animals were reviewed and approved by the cantonal veterinary authority (authorization number VD1890.2).
RG and JS-D designed and RG imaged the demo sample. NC and OB wrote the Warpy code. PH wrote the Image Combiner Warpy code. NC performed the data analysis. NC, PH, AS and RG wrote the manuscript.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that m ay be made by its manufacturer, is not guaranteed or endorsed by the publisher.
The histology data was prepared at the EPFL Histology Core Facility. We thank Nathalie Müller and Gian-Filippo Mancini for the preparation of the samples. We thank the Fiji and QuPath open source community for the continuous support they offer, especially Pete Bankhead, John Bogovic, Christian Tischer, Jan Eglinger and Curtis Rueden.
The Supplementary Material for this article can be found online at: