One of the main drawbacks of optical microscopes is their limitations in resolving finer details. Scanning electron microscopes (SEM) use electron imaging, and the wavelength of electrons is much smaller than that of photons, easily achieving magnifications of tens or even hundreds of thousands of times. However, SEM can only obtain 2D images and cannot directly generate 3D shapes. Compared to 2D images, three-dimensional structures can provide more information for many micro research fields. For example, SEM has important applications in the petroleum field, (Zhu, 2013; Liu et al., 2023; Wu et al., 2023), used SEM in the analysis of the micro pore structure of reservoir rocks. Yao used SEM combined with deep learning based image processing to characterize the nanoscale pore structure of rock debris (Yao et al., 2022). Scholars have also used SEM to observe and analyze the microstructure of solid/oil well cement stone (Li et al., 2016; Zhang et al., 2013; Song et al., 2017; Song et al., 2020). Munawar used optical polarization microscopy in transmission mode to document the whole-rock mineralogy, diagenetic relationships, porosity characteristics and clay occurrences in the pores of these samples (Munawar et al., 2018a). Wang used focused ion beam scanning electron microscopy (FIB-SEM) to reconstruct the internal organic matter pores of shale using a three-dimensional-slicing-imaging reconstruction technology route, but its literature also indicated that FIB can cause damage to the sample during the slicing process, which is a destructive analysis method (Wang et al., 2019). Munawar mentions that micro-CT imaging has certainly some limitations concerning sample size and resolution (Munawar et al., 2018b). It is reliable to captures micron scale features, whereas it is not possible for micro-CT to capture nanometer scale features (Munawar et al., 2018b; Luo et al., 2022; Ji et al., 2015). There are scholars used X-ray CT to perform three-dimensional measurements on carbonate rock and sandstone sample (Munawar et al., 2021). A series of two-dimensional slices stack into a 3D image. Data are arranged in an array of pixels in the two-dimensional slice. A pixel in the third dimension makes it three-dimensional volume which is a voxel. This is, the distance between two consecutive slices is a voxel. The 3D model formed by stacking sliced images is a discrete model, but compared to FIBSEM, this is a reconstruction method for non-destructive samples. The resolution of the new scanning electron microscope can reach 1 nm. Based on SEM 3D reconstruction, nanoscale, spatially continuous 3D models can be obtained, which is helpful for calculating porosity. There are many scholars also use SEM equipment in petroleum energy research (Huang et al., 2023a; Huang et al., 2023b; Tan et al., 2023; Tan et al., 2020; Huang et al., 2020; Huang et al., 2022). SEM can provide high-resolution images without damaging the sample. If standard SEM machines can be used to restore the microscopic three-dimensional structure of samples such as rocks in the above research, it will provide more information for analyzing pore structure, material toughness, etc. without damage. In order to effectively measure and visualize the three-dimensional structure of microscopic samples, many scholars have been trying to convert scanning electron microscopy into microscopic three-dimensional measurement tools.

Currently, the commonly used 3D reconstruction algorithms under scanning electron microscopy include the Photometric Stereo (PS) method (Ikeuchi, 1981; Basri et al., 2007; Goldman et al., 2010). As shown in Figure 1, in the PS based method, the sample stage and detector of the scanning electron microscope remain stationary, providing illumination in different directions to the sample stage. From a single perspective, the detector captures a set of 2D images in different lighting directions and quickly calculates the 3D geometric shape of the SEM sample. However, since only one perspective is used, it is not possible to create a complete point clouds, and the PS method requires illumination from different directions on the sample stage, making it difficult to have such additional lighting conditions using standard scanning electron microscopy machines, which increases equipment costs and limitations.

(Wang et al., 2013) used SFS (shape from shading) technology to analyze the grayscale information of a single top view SEM image, and ultimately reconstructed the three-dimensional surface structure. However, due to the fixed perspective, a complete point clouds cannot be obtained.

(Ding et al., 2019) adopted a nanorobot system to automatically capture a set of scanning electron microscope images along a linear path with a fixed step size, and used light field theory to reconstruct the three-dimensional surface model of SEM samples. As shown in Figure 2A, the sample is fixed on the sample stage via either glue, tape or customized grippers according to sample properties. After assembling the nano-robotic manipulation system inside SEM, the linear positioner is utilized to move step by step along x-axis with a fixed step size, and one SEM image is taken after each step. Figure 2B shows the SEM images obtained at each step. However, not all SEM devices are equipped with nanorobot systems, and the step accuracy of nanorobot systems is limited, requiring a large number of images to be captured.

Tafti et al., 2015; Tafti et al., 2016 designed an optimized, parameter adaptive, and intelligent multi view method, 3DSEM++, for 3D surface reconstruction of SEM images. He also publicly and freely provided a 3D SEM image set, which is derived from Tafti’s publicly available image set. Some scholars have studied the use of optical measurement software for three-dimensional reconstruction of SEM images (Eulitz and Reiss, 2015; Kareiva et al., 2015), but optical measurement software often does not consider the imaging characteristics of SEM itself, and generally requires images to contain EXIF information and camera model, which SEM images often do not have.

The parallax depth mapping method (Samak et al., 2007; Baghaie et al., 2017; Yi et al., 2019; Bian, 2020) is the simplest and fastest way to obtain the three-dimensional structure of SEM samples. As shown in Figure 3A, θ is the rotation angle, and the relationship between disparity d and depth h is given by a formula.

h = d 2 * ⁡ sin ⁡ θ , by rotating the sample stage at a certain angle to obtain two SEM images, a point clouds can be created based on parallax. However, since parallax depth mapping can only use two images at once, it is also not possible to create a complete point clouds. In response to the insufficient integrity of the point clouds reconstructed by the disparity depth mapping method, Rehman used disparity depth mapping combined with point clouds registration to achieve multi view SEM 3D reconstruction [Rehman, 2018]. As shown in Figure 3B, multiple point clouds are obtained by using the parallax depth mapping multiple times, and then point clouds registration is performed

Liu used incremental motion recovery structure (SFM) to perform 3D reconstruction of the SEM image sequence of the Drosophila head, and successfully restored the point clouds (Liu, 2018). SFM uses singular value decomposition of the essential matrix to obtain initial rotation and translation parameters. Unfortunately, this method yields poor results on the image set presented in this paper.

This article improves the shortcomings of the aforementioned SEM based 3D reconstruction methods, such as a single perspective, equipment cost limitations, lack of consideration for SEM imaging characteristics, and time consumption. It proposes a method for restoring the detector pose, which is different from traditional essential matrix singular value decomposition. It uses imaging equations of multiple image points to form a system of equations, analyzes the known internal and external parameters in the imaging model, and calculates the motion parameters of the position. Triangulation is used to generate point clouds of different parts of the microscopic sample, and then point clouds registration is combined to obtain a more complete point clouds. Thus, standard SEM machines can be used to achieve three-dimensional reconstruction of microscopic samples from multiple perspectives.

The content of this article is arranged as follows. The first section is an introduction, which introduces the commonly used methods and shortcomings of three-dimensional reconstruction of microscopic samples under SEM.

Section 2 will introduce the publicly available 3D SEM image set and point out that there are some known conditions in this image set that are beneficial for subsequent calibration work.

Section 3 will introduce the SEM three-dimensional measurement model. When the magnification of the SEM exceeds 1,000, the SEM imaging process is similar to a double telecentric lens. At this point, the parallel projection model is more suitable for the SEM imaging model. Then, based on the publicly available 3D SEM image set, a motion parameter estimate method between SEM image sequences suitable for this image set is derived.

Section 4 will present a more specific experimental process and results.

In Section 5, the evaluation indicators for point clouds registration, including root mean square error and relative rotation error, will be introduced. Due to the fact that the relative translation motion in relative transformation does not have a true value as a reference, relative translation error cannot be used as an evaluation indicator. Under these indicators, the registration results of our method will be compared and analyzed with those of the disparity depth mapping method.

Section 6 will provide the discussion and conclusion of this article.

The 3D SEM image set used in this article is from the publicly available SEM image set by Tafti, available at the website https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/HVBW0Q#__sid=js0, the characteristic of this image set is that, as shown in Figure 3A, in the image sequence, the Y-axis of the sample stage itself is used as the rotation axis, and a fixed angle is rotated around that axis θ to obtain the next image, during the actual shooting process, this rotation operation will be continuously performed to obtain multiple SEM images. The rotation angle θ will serve as an important rotational motion parameter in the imaging model.

Although the actual operation process involves the sample stage rotating around its Y-axis while the imaging plane remains stationary, it can be seen that the sample stage remains stationary and the imaging plane rotates around the sample stage because the motion is relative. Figure 3B shows the equivalent shooting process of SEM images of “ Pollen grain from Brassica rapa ” in the image set.

This publicly available 3DSEM image set was obtained using standard SEM equipment, and its special motion method is to apply the disparity depth mapping method. Many scholars have also used this image set to conduct extensive research based on disparity depth mapping. Among them, Rehman added point clouds registration to it, enabling all SEM images to be used and obtaining a more complete point clouds (Rehman, 2018). However, the above studies are all based on disparity depth mapping. In fact, this image set can use calibration methods to estimate motion parameters, and then use triangulation to recover point clouds. Section 5 will compare these two methods.

Although parallax depth mapping and SFM can also be applied on standard SEM devices, the point clouds registration method after parallax depth mapping requires the sample stage to perform a special rotation motion as shown in Figure 3A multiple times in a row, which limits the shooting angle of the SEM device and the point clouds registration error is large and unstable. However, the method proposed in this paper only requires one special rotation motion as shown in Figure 3A to obtain the initial point clouds, the rotation and translation parameters of subsequent images can be estimated using the PnP algorithm, so the shooting angle of subsequent images is arbitrary and unrestricted. At the same time, the method proposed in this paper is based on point clouds obtained through triangulation, which can exclude some error points and noise points through the indicator of reprojection error, which is an indicator that disparity depth mapping does not have.

The difference between this method and SFM lies in the approach of obtaining initial three-dimensional points. SFM uses singular value decomposition of the essential matrix to estimate the initial rotation and translation parameters, thereby obtaining a projection matrix to solve the three-dimensional points. The rotation and translation parameters of subsequent images are also estimated using the PnP algorithm. SFM does not require SEM equipment to perform special movements on Figure 3A, Unfortunately, on publicly available 3D SEM datasets, the rotation parameters estimated through the essential matrix are far from the actual rotation parameters, making it impossible to obtain good initial point clouds. Fortunately, SEM equipment is an indoor device that can control rotation motion. Compared to SFM estimated rotation parameters, using known rotation parameters is more stable. Therefore, the method proposed in this article only requires one special motion as shown in Figure 3A, after obtaining the real rotation parameters, the translation parameters can be estimated by constructing an equation system through the imaging model, and then the initial point clouds can be obtained through triangulation. The rotation and translation parameters of subsequent images can also be estimated using the PnP algorithm. In fact, the method proposed in this article is a compromise between disparity depth mapping and SFM. We believe that the proposed method can replace the disparity depth mapping and SFM.