Vertebrate paleontology is based on fossils that are remains of ancient organisms. Because fossils usually do not preserve molecular or behavior information, paleontologists mainly focus on their morphology, which not only includes exterior features but also interior structures like brain endocasts and inner ears. Traditionally, researchers used destructive thin sectioning to reveal interior structures, which totally destroys the fossils. With the application of non-destructive 3D imaging techniques, like CT (computed tomography) scan and synchrotron radiation scanning, paleontologists can observe and interact with previously hidden structures without making damages. CT scan and other non-destructive imaging techniques have greatly facilitated the development of vertebrate paleontology not only in revealing hidden structures but also providing 3D models for teaching and exhibition.

Recently, CT scans have been successfully applied to various branches in vertebrate paleontology (Figures 1A–E), for example the studies of archosaur brain endocasts (Watanabe et al., 2019), mammalian inner ears (Luo and Ketten 1991), theropod dinosaur body mass estimation (Allen et al., 2013), finite element analysis of the Allosaurus skull (Rayfield et al., 2001), and horned dinosaur tooth replacement patterns (He et al., 2018). Figure 1F illustrates a general workflow in fossil CT scan processing from original fossil to unsegmented volumes, then to individually segmented bones. For fragile specimens that cannot be mechanically prepared, CT scan (or other non-invasive scanning techniques) is the only way to make detailed observations. Such scanners are now common elements in many paleontological laboratories. However, processing the data generated by CT and other observational equipment is both arduous and time-consuming.

Current CT data processing is laborious, especially for morphologically complex objects like human bodies and vertebrate fossils. CT scanners differentiate volumes by their absorptions of X-ray radiation, which is primarily controlled by their densities. However, fossils and their surrounding matrices are often similar or even identical in densities (Figures 2B,C), making the differentiation between matrix and specimen difficult. Many researchers have to rely on manual segmentation, sometimes slice by slice, according to their own judgment and understanding of anatomy. One of the major flaws in current CT processing is the lack of reproducibility. Most reconstruction have to be based on anatomical knowledge and professional experience.

Most paleontological CT data are processed in commercial software such as VGstudio by Volume Graphics and Mimics by Materialise. There is also a handful of opensource software (e.g., ImageJ developed by the National Institute of Health, United States) but most of them provide users with less functionality comparing to commercial ones. A lot of 3D image processing software offers thresholding and region growing functions for more efficient segmentation. Thresholding differentiates voxels according to their grey values, therefore it is functionless when ROI (region of interest) and surrounding areas are similar in density. Region growing requires a seed voxel to grow from in given directions following a grey value range, which means all connected voxels within a given grey value range will be labeled. Thresholding and region growing work best when there is significant contrast between ROI and surrounding areas, and their boundaries can be clearly defined. Although current software can process multiple slices at the same time based on linear interpolation, it is often insufficient for more rapid and accurate segmentation, especially of intricately preserved fossils. In CT data processing, if there are significant grey value differences (which represent densities) between target elements (bony fossils, stained soft tissue, and cavity structures) and undesired elements (i.e., rock matrices or surrounding tissues), it is easy to segment. But due to the complex taphonomy, fragmentary preservation, similarity between rocks and embedded fossils, CT data processing is usually exhausting. Figures 2A,B show two example slices of fossil protoceratopsians segmented by manual labelling, U-net, region growing, and thresholding. While manual labelling outperforms all other methods, it costs much more time and labor especially facing high-resolution scans, and either thresholding or region growing cannot properly segment the desired regions therefore requires subsequent manual processing. Details of segmentation maps in Figure 2C indicates the complexity (e.g. tooth surrounded by dentary bone matrix) in fossil CT scans. In conclusion, there exist three major drawbacks in current 3D imaging processing:

1. Manual processing is necessary when ROI and surrounding areas are similar in densities or their boundaries are ambiguous.

Over the last decade, deep learning has shown incredible potential when applied in complicated tasks including image processing. Since 2013, automated processing (classification, object detection, and segmentation) of medical images has utilized by deep learning (Ker et al., 2017; Litjens et al., 2017; Shen et al., 2017; Cardoso et al., 2017; Roth et al., 2018; Ting et al., 2018). Many image-related tasks in medical sciences can be effectively accomplished by deep learning algorithms such as brain lesion segmentation (Kamnitsas et al., 2017), nodule classification (Ciompi et al., 2017), skin cancer detection (Esteva et al., 2017), etc. Considering the similarity between medical and vertebrate paleontological CT images, it is natural to apply deep learning to fossil data. However, there are three major differences between fossil and medical CT scans:

1. To reduce the harm to patients, medical scans are usually restricted to relatively low energy and short exposure time. Therefore, they cannot produce as much contrast as the higher energy beams used for fossils. Also, paleontological scan data usually have higher resolutions, thus requires greater computational capability.

In this study, we generate a dataset from the CT-scans of three well-preserved protoceratopsian skulls (Ornithischia, Dinosauria) from Gobi Desert, Mongolia, and manually annotated bone structures in each slice for subsequent analysis. We test the performance of classic U-net (Ronneberger et al., 2015) and DeepLab v3+ (Chen et al., 2016) with various modifications (Figure 3).

In this study, we first test the performance of classic U-net for image segmentation (Ronneberger et al., 2015, Figure 3A), and then modified DeepLab v3+ (Chen et al., 2016). We use MobileNet v1 (Howard et al., 2017) or ResNet v2 (He et al., 2016) in the encoder, which introduces separable convolutional layers with depthwise and pointwise layers followed by BatchNorm and ReLU layer, respectively (Figure 3B&C), thus to reduce the computational cost and number of parameters but keep most of the performance. The skip connections between Astrous Spatial Pyramid Pooling (ASPP) and decoders allow the recognition of features in different scale, thus to better capture patterns that distinguishing fossils and rock matrices. The Deeplab v3+ model structures are shown in Figure 3C and Table 2.

Part of the segmentation results are shown in Figure 4. Classic U-net model reached a high accuracy of 98.09 and 97.44% in test and validation dataset, however, the cross-entropy loss for both datasets are relatively considerable of 4.94 and 9.92%, respectively. Segmentation results from varieties of modified DeepLab v3+ reaches mean dice between 0.738 and 0.894, and mean IOU (intersection over union) between 0.612 and 0.817. The Sørensen–Dice coefficient and IOU scores are calculated as following: I O U = S e g m e n t a t i o n Re s u l t ∩ G r o u n d T r u t h S e g m e n t a t i o n Re s u l t ∪ G r o u n d T r u t h , I O U ∈ [ 0,1 ] 1 d i c e = 1 2 × ( 1 Pr e c i s i o n + 1 Re c a l l ) Therefore, d i c e = 2 T P 2 T P + F P + F N where TP represents true positive, FP false positive, and FN false negative, respectively. The DeepLab v3+ model with MobileNet v1 as feature extractor and two skip connections outperformed other models. The prediction maps are shown in Figure 4B.

The fast advancements in techniques not only enable unprecedented resolution in observation of fossil material, but also increase the cost in data processing. Currently paleontologists are spending days to weeks in segmenting fossil scans, the introduction of deep learning can reduce that time to minutes. Although the 3D renderings from automated segmented slices are not as meticulous as manual results (Figure 5), the comparison between raw reconstruction and deep learning segmented volumes has shown that deep learning can at least reduce processing time in differentiating fossils and rock matrices.

To further test the generalization performance of deep neural network, we use the trained best-performed Deeplab v3+ network to segment other vertebrate fossils from the Gobi Desert, Mongolia with similar sedimentary environments, which should reduce the bias from sampling to the minimum. The new dataset (see supplementary material) including CT-scanned fossil slices of ornithischian dinosaur Haya (IGM 100-3178, IGM 100-3181) and Pinacosaurus (IGM 100-3186). However, these non-protoceratopsian fossil images are extremely poorly segmented (Figure 6). Although both U-net and modified DeepLab v3+ shows close to manual segmentation performance in at least part of the protoceratopsian dataset, the capability of extrapolation on other datasets seems to be largely limited. If we are applying deep learning in future paleontological studies (e.g., facilitate CT segmentation), generalization performance is a must. The Gobi Desert in Mongolia has yielded a wide range of both vertebrate and invertebrate fossils, it cannot be expected to have a training dataset comprising fossil data from all known taxa. On the other hand, since our training dataset is solely derived from protoceratopsian dinosaurs, more specifically from specimens in early developmental stages, it is likely to have intrinsic bias in the dataset due to the poor ossification status of bones and other possible factors. Such overfitting problem is a major concern in deep learning studies. Comparing to medical applications that focus on a particular structure (e.g., brain lesion Kamnitsas et al., 2017) mentioned before, fossil segmentation tasks in paleontological studies seem to be more non-specific, therefore, the models constructed in this study may improve by feeding more generalized training data, possibly covering more taxa and developmental stages. It should be also noted that in this study, we only performed semantic segmentation on fossil bone without distinguishing different structures, but the grey value patterns are obviously varied across different tissues like teeth and dentary bones (Figure 2C).

Deep learning has already shown its power in many research branches such as Go game playing, medical imaging processing, and earth science studies (Silver et al., 2016; Esteva et al., 2017; Litjens et al., 2017; Reichstein et al., 2019; Silver et al., 2017; Bergen et al., 2019). The recent advancements in observation techniques and increasing available fossil specimens facilitate paleontological data growing in an exponential manner, which enable large-scale data-driven paleontological and earth science project, for example the Deep-time Digital Earth (DDE, Wang et al., 2021). There have been only few studies introducing deep learning in the field of paleontology until recently, for example the segmentation of fish fossil bones (Hou et al., 2020; Hou et al., 2021), planktonic foraminiferal species identification (Hsiang et al., 2019), and automated pollen recognition (Bourel et al., 2020). The studied materials cover plants, invertebrates, and vertebrates distributed in different ages, indicating the potential to apply deep learning in various branches of paleontological studies. Including this study, most of paleontological deep learning studies focus on faster and automated imaging classification and segmentation, other potential application scenarios in paleontology are less investigated, for example digitally reconstruction of incomplete fossils based on GAN (generative adversarial network) and data mining from published literatures to reconstruct macro evolutionary patterns. With more available paleontological and maturation of algorithms, we shall expect wider applications of deep learning in paleontological studies soon.